wehub-resource-sync
100 lines
2.7 KiB
Plaintext
100 lines
2.7 KiB
Plaintext
---
|
|
title: "Crawl a URL using Firecrawl"
|
|
sidebarTitle: "Firecrawl URL crawl"
|
|
description: "This example demonstrates how to crawl a URL using Firecrawl with Trigger.dev."
|
|
---
|
|
|
|
## Overview
|
|
|
|
Firecrawl is a tool for crawling websites and extracting clean markdown that's structured in an LLM-ready format.
|
|
|
|
Here are two examples of how to use Firecrawl with Trigger.dev:
|
|
|
|
## Prerequisites
|
|
|
|
- A project with [Trigger.dev initialized](/quick-start)
|
|
- A [Firecrawl](https://firecrawl.dev/) account
|
|
|
|
## Example 1: crawl an entire website with Firecrawl
|
|
|
|
This task crawls a website and returns the `crawlResult` object. You can set the `limit` parameter to control the number of URLs that are crawled.
|
|
|
|
```ts trigger/firecrawl-url-crawl.ts
|
|
import Firecrawl from "@mendable/firecrawl-js";
|
|
import { task } from "@trigger.dev/sdk";
|
|
|
|
// Initialize the Firecrawl client with your API key
|
|
const firecrawlClient = new Firecrawl({
|
|
apiKey: process.env.FIRECRAWL_API_KEY, // Get this from your Firecrawl dashboard
|
|
});
|
|
|
|
export const firecrawlCrawl = task({
|
|
id: "firecrawl-crawl",
|
|
run: async (payload: { url: string }) => {
|
|
const { url } = payload;
|
|
|
|
// Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
|
|
const crawlResult = await firecrawlClient.crawl(url, {
|
|
limit: 100, // Limit the number of URLs to crawl
|
|
scrapeOptions: {
|
|
formats: ["markdown", "html"],
|
|
},
|
|
});
|
|
|
|
if (crawlResult.status === "failed") {
|
|
throw new Error(`Failed to crawl: ${url}`);
|
|
}
|
|
|
|
return {
|
|
data: crawlResult,
|
|
};
|
|
},
|
|
});
|
|
```
|
|
|
|
### Testing your task
|
|
|
|
You can test your task by triggering it from the Trigger.dev dashboard.
|
|
|
|
```json
|
|
"url": "<url-to-crawl>" // Replace with the URL you want to crawl
|
|
```
|
|
|
|
## Example 2: scrape a single URL with Firecrawl
|
|
|
|
This task scrapes a single URL and returns the `scrapeResult` object.
|
|
|
|
```ts trigger/firecrawl-url-scrape.ts
|
|
import Firecrawl from "@mendable/firecrawl-js";
|
|
import { task } from "@trigger.dev/sdk";
|
|
|
|
// Initialize the Firecrawl client with your API key
|
|
const firecrawlClient = new Firecrawl({
|
|
apiKey: process.env.FIRECRAWL_API_KEY, // Get this from your Firecrawl dashboard
|
|
});
|
|
|
|
export const firecrawlScrape = task({
|
|
id: "firecrawl-scrape",
|
|
run: async (payload: { url: string }) => {
|
|
const { url } = payload;
|
|
|
|
// Scrape: scrapes a URL and get its content in LLM-ready format (markdown, structured data via LLM Extract, screenshot, html)
|
|
const scrapeResult = await firecrawlClient.scrape(url, {
|
|
formats: ["markdown", "html"],
|
|
});
|
|
|
|
return {
|
|
data: scrapeResult,
|
|
};
|
|
},
|
|
});
|
|
```
|
|
|
|
### Testing your task
|
|
|
|
You can test your task by triggering it from the Trigger.dev dashboard.
|
|
|
|
```json
|
|
"url": "<url-to-scrape>" // Replace with the URL you want to scrape
|
|
```
|