wehub-resource-sync
c56bef871b
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
109 lines
4.1 KiB
Plaintext
109 lines
4.1 KiB
Plaintext
---
|
|
title: "FirecrawlCrawler"
|
|
id: firecrawlcrawler
|
|
slug: "/firecrawlcrawler"
|
|
description: "Use Firecrawl to crawl websites and return the content as Haystack Documents. Unlike single-page fetchers, FirecrawlCrawler follows links and discovers subpages."
|
|
---
|
|
|
|
# FirecrawlCrawler
|
|
|
|
Use Firecrawl to crawl websites and return the content as Haystack Documents. Unlike single-page fetchers, FirecrawlCrawler follows links and discovers subpages.
|
|
|
|
<div className="key-value-table">
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Most common position in a pipeline** | In indexing or query pipelines as the data fetching step |
|
|
| **Mandatory run variables** | `urls`: A list of URLs (strings) to start crawling from |
|
|
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) |
|
|
| **API reference** | [Firecrawl](/reference/integrations-firecrawl) |
|
|
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/firecrawl |
|
|
|
|
</div>
|
|
|
|
## Overview
|
|
|
|
`FirecrawlCrawler` uses [Firecrawl](https://firecrawl.dev) to crawl one or more URLs and return the extracted content as Haystack `Document` objects. Starting from each given URL, it follows links to discover subpages up to a configurable limit. This makes it well-suited for ingesting entire websites or documentation sites, not just single pages.
|
|
|
|
Firecrawl returns content in a structured format that works well as input for LLMs. Each crawled page becomes a separate `Document` with the page content in the `content` field and metadata, such as title, URL, and description, in the `meta` field.
|
|
|
|
### Crawl parameters
|
|
|
|
You can control the crawl behavior through the `params` argument. Some commonly used parameters:
|
|
|
|
- `limit`: Maximum number of pages to crawl per URL. Defaults to `1`. Without a limit, Firecrawl may crawl all subpages and consume credits quickly.
|
|
- `scrape_options`: Controls the output format. Defaults to `{"formats": ["markdown"]}`.
|
|
|
|
See the [Firecrawl API reference](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post) for the full list of available parameters.
|
|
|
|
### Authorization
|
|
|
|
`FirecrawlCrawler` uses the `FIRECRAWL_API_KEY` environment variable by default. You can also pass the key explicitly at initialization:
|
|
|
|
```python
|
|
from haystack.utils import Secret
|
|
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
|
|
|
crawler = FirecrawlCrawler(api_key=Secret.from_token("<your-api-key>"))
|
|
```
|
|
|
|
To get an API key, sign up at [firecrawl.dev](https://firecrawl.dev).
|
|
|
|
### Installation
|
|
|
|
Install the Firecrawl integration with:
|
|
|
|
```shell
|
|
pip install firecrawl-haystack
|
|
```
|
|
|
|
## Usage
|
|
|
|
### On its own
|
|
|
|
```python
|
|
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
|
|
|
crawler = FirecrawlCrawler(params={"limit": 3})
|
|
|
|
result = crawler.run(urls=["https://docs.haystack.deepset.ai/docs/intro"])
|
|
documents = result["documents"]
|
|
|
|
for doc in documents:
|
|
print(f"{doc.meta.get('title')} - {doc.meta.get('url')}")
|
|
```
|
|
|
|
### In a pipeline
|
|
|
|
Below is an example of an indexing pipeline that uses `FirecrawlCrawler` to crawl a documentation site and store the results in an `InMemoryDocumentStore`.
|
|
|
|
```python
|
|
from haystack import Pipeline
|
|
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
|
from haystack.components.preprocessors import DocumentSplitter
|
|
from haystack.components.writers import DocumentWriter
|
|
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
|
|
|
document_store = InMemoryDocumentStore()
|
|
|
|
crawler = FirecrawlCrawler(params={"limit": 10})
|
|
splitter = DocumentSplitter(split_by="sentence", split_length=5)
|
|
writer = DocumentWriter(document_store=document_store)
|
|
|
|
indexing_pipeline = Pipeline()
|
|
indexing_pipeline.add_component("crawler", crawler)
|
|
indexing_pipeline.add_component("splitter", splitter)
|
|
indexing_pipeline.add_component("writer", writer)
|
|
|
|
indexing_pipeline.connect("crawler.documents", "splitter.documents")
|
|
indexing_pipeline.connect("splitter.documents", "writer.documents")
|
|
|
|
indexing_pipeline.run(
|
|
data={
|
|
"crawler": {
|
|
"urls": ["https://docs.haystack.deepset.ai/docs/intro"],
|
|
},
|
|
},
|
|
)
|
|
```
|