chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-fetchers
|
||||
slug: "/external-integrations-fetchers"
|
||||
description: "External integrations that enable data extraction from different sources."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable data extraction from different sources.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [Apify](https://haystack.deepset.ai/integrations/apify) | Extract data from e-commerce websites, social media platforms (such as Facebook, Instagram, and TikTok), search engines, online maps, and more, while automating web tasks. |
|
||||
| [Bright Data](https://haystack.deepset.ai/integrations/bright-data) | Extract data from 45+ websites, get search engine results, and access geo-restricted content using Bright Data's web scraping services. |
|
||||
| [Mastodon](https://haystack.deepset.ai/integrations/mastodon-fetcher) | Fetch a Mastodon username's latest posts. |
|
||||
| [Notion](https://haystack.deepset.ai/integrations/notion-extractor) | Extract pages from Notion to Haystack Documents. |
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "FirecrawlCrawler"
|
||||
id: firecrawlcrawler
|
||||
slug: "/firecrawlcrawler"
|
||||
description: "Use Firecrawl to crawl websites and return the content as Haystack Documents. Unlike single-page fetchers, FirecrawlCrawler follows links and discovers subpages."
|
||||
---
|
||||
|
||||
# FirecrawlCrawler
|
||||
|
||||
Use Firecrawl to crawl websites and return the content as Haystack Documents. Unlike single-page fetchers, FirecrawlCrawler follows links and discovers subpages.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing or query pipelines as the data fetching step |
|
||||
| **Mandatory run variables** | `urls`: A list of URLs (strings) to start crawling from |
|
||||
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) |
|
||||
| **API reference** | [Firecrawl](/reference/integrations-firecrawl) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/firecrawl |
|
||||
| **Package name** | `firecrawl-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FirecrawlCrawler` uses [Firecrawl](https://firecrawl.dev) to crawl one or more URLs and return the extracted content as Haystack `Document` objects. Starting from each given URL, it follows links to discover subpages up to a configurable limit. This makes it well-suited for ingesting entire websites or documentation sites, not just single pages.
|
||||
|
||||
Firecrawl returns content in a structured format that works well as input for LLMs. Each crawled page becomes a separate `Document` with the page content in the `content` field and metadata, such as title, URL, and description, in the `meta` field.
|
||||
|
||||
### Crawl parameters
|
||||
|
||||
You can control the crawl behavior through the `params` argument. Some commonly used parameters:
|
||||
|
||||
- `limit`: Maximum number of pages to crawl per URL. Defaults to `1`. Without a limit, Firecrawl may crawl all subpages and consume credits quickly.
|
||||
- `scrape_options`: Controls the output format. Defaults to `{"formats": ["markdown"]}`.
|
||||
|
||||
See the [Firecrawl API reference](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post) for the full list of available parameters.
|
||||
|
||||
### Authorization
|
||||
|
||||
`FirecrawlCrawler` uses the `FIRECRAWL_API_KEY` environment variable by default. You can also pass the key explicitly at initialization:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
||||
|
||||
crawler = FirecrawlCrawler(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get an API key, sign up at [firecrawl.dev](https://firecrawl.dev).
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Firecrawl integration with:
|
||||
|
||||
```shell
|
||||
pip install firecrawl-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
||||
|
||||
crawler = FirecrawlCrawler(params={"limit": 3})
|
||||
|
||||
result = crawler.run(urls=["https://docs.haystack.deepset.ai/docs/intro"])
|
||||
documents = result["documents"]
|
||||
|
||||
for doc in documents:
|
||||
print(f"{doc.meta.get('title')} - {doc.meta.get('url')}")
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of an indexing pipeline that uses `FirecrawlCrawler` to crawl a documentation site and store the results in an `InMemoryDocumentStore`.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
crawler = FirecrawlCrawler(params={"limit": 10})
|
||||
splitter = DocumentSplitter(split_by="sentence", split_length=5)
|
||||
writer = DocumentWriter(document_store=document_store)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("crawler", crawler)
|
||||
indexing_pipeline.add_component("splitter", splitter)
|
||||
indexing_pipeline.add_component("writer", writer)
|
||||
|
||||
indexing_pipeline.connect("crawler.documents", "splitter.documents")
|
||||
indexing_pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
indexing_pipeline.run(
|
||||
data={
|
||||
"crawler": {
|
||||
"urls": ["https://docs.haystack.deepset.ai/docs/intro"],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: "GoogleDriveFetcher"
|
||||
id: googledrivefetcher
|
||||
slug: "/googledrivefetcher"
|
||||
description: "Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams."
|
||||
---
|
||||
|
||||
# GoogleDriveFetcher
|
||||
|
||||
Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), before a Router or File Converters |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` <br /> <br />`targets`: A list of `Document`s (from `GoogleDriveRetriever`) or raw Google Drive file ids / URLs |
|
||||
| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content |
|
||||
| **API reference** | [Google Drive](/reference/integrations-google-drive) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive |
|
||||
| **Package name** | `google-drive-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleDriveFetcher` downloads the full content of Google Drive files through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3) and returns `ByteStream` objects, ready for a downstream converter.
|
||||
|
||||
It complements [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), which returns only metadata (and optionally exported text). Wire the retriever's `documents` (or a list of file ids / Drive URLs) into the fetcher to download the underlying content. The fetcher dispatches on each file's mime type:
|
||||
|
||||
- **Binary files** (PDF, DOCX, images, ...) are downloaded as-is via `files.get?alt=media`.
|
||||
- **Native Google Docs/Sheets/Slides** are exported with `files.export`, by default to the Office formats (DOCX/XLSX/PPTX), configurable via `export_mime_types`.
|
||||
- **Folders** and other non-downloadable Google types (Forms, Sites, ...) are skipped.
|
||||
|
||||
Each `ByteStream`'s `meta` carries `file_id`, `web_url`, `file_name`, and `content_type`. Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`XLSXToDocument`](../converters/xlsxtodocument.mdx), or [`PPTXToDocument`](../converters/pptxtodocument.mdx)).
|
||||
|
||||
### Authentication
|
||||
|
||||
The fetcher takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows reading file content, for example `https://www.googleapis.com/auth/drive.readonly`. Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.
|
||||
|
||||
### Error handling and concurrency
|
||||
|
||||
- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the file is skipped, so the remaining files are still returned.
|
||||
- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors.
|
||||
- `max_concurrent_requests` (default `5`): bounds the number of files fetched concurrently by `run_async` to avoid tripping Drive rate limits. It has no effect on the synchronous `run`, which fetches files one at a time.
|
||||
- `export_mime_types`: overrides the default native-Google-to-Office export mapping. Drive caps a single export at 10 MB.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Google Drive integration with:
|
||||
|
||||
```shell
|
||||
pip install google-drive-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
`access_token` below is a per-user delegated Google OAuth bearer token. You can pass either raw file ids / Drive URLs or the `Document`s produced by `GoogleDriveRetriever`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher
|
||||
|
||||
fetcher = GoogleDriveFetcher()
|
||||
|
||||
result = fetcher.run(
|
||||
access_token="my-delegated-google-token",
|
||||
targets=[
|
||||
"https://drive.google.com/file/d/1AbCdEfGhIjKlMnOpQrStUvWxYz/view",
|
||||
],
|
||||
)
|
||||
|
||||
for stream in result["streams"]:
|
||||
print(stream.meta["file_name"], stream.meta["content_type"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) searches Drive, `GoogleDriveFetcher` downloads the matching files, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.routers import FileTypeRouter
|
||||
from haystack.components.converters import PyPDFToDocument, DOCXToDocument
|
||||
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
from haystack_integrations.components.retrievers.google_drive import (
|
||||
GoogleDriveRetriever,
|
||||
)
|
||||
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"resolver",
|
||||
OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://oauth2.googleapis.com/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"),
|
||||
scopes=["https://www.googleapis.com/auth/drive.readonly"],
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5))
|
||||
pipeline.add_component("fetcher", GoogleDriveFetcher())
|
||||
pipeline.add_component(
|
||||
"router",
|
||||
FileTypeRouter(
|
||||
mime_types=[
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
],
|
||||
),
|
||||
)
|
||||
pipeline.add_component("pdf_converter", PyPDFToDocument())
|
||||
pipeline.add_component("docx_converter", DOCXToDocument())
|
||||
|
||||
# The same token feeds both the retriever and the fetcher.
|
||||
pipeline.connect("resolver.access_token", "retriever.access_token")
|
||||
pipeline.connect("resolver.access_token", "fetcher.access_token")
|
||||
|
||||
# The retrieved documents become the fetcher's targets.
|
||||
pipeline.connect("retriever.documents", "fetcher.targets")
|
||||
|
||||
# Route each fetched ByteStream to the matching converter.
|
||||
pipeline.connect("fetcher.streams", "router.sources")
|
||||
pipeline.connect("router.application/pdf", "pdf_converter.sources")
|
||||
pipeline.connect(
|
||||
"router.application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"docx_converter.sources",
|
||||
)
|
||||
|
||||
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
|
||||
```
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
title: "LinkContentFetcher"
|
||||
id: linkcontentfetcher
|
||||
slug: "/linkcontentfetcher"
|
||||
description: "With LinkContentFetcher, you can use the contents of several URLs as the data for your pipeline. You can use it in indexing and query pipelines to fetch the contents of the URLs you give it."
|
||||
---
|
||||
|
||||
# LinkContentFetcher
|
||||
|
||||
With LinkContentFetcher, you can use the contents of several URLs as the data for your pipeline. You can use it in indexing and query pipelines to fetch the contents of the URLs you give it.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing or query pipelines as the data fetching step |
|
||||
| **Mandatory run variables** | `urls`: A list of URLs (strings) |
|
||||
| **Output variables** | `streams`: A list of [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **API reference** | [Fetchers](/reference/fetchers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/fetchers/link_content.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`LinkContentFetcher` fetches the contents of the `urls` you give it and returns a list of content streams. Each item in this list is the content of one link it successfully fetched in the form of a `ByteStream` object. Each of these objects in the returned list has metadata that contains its content type (in the `content_type` key) and its URL (in the `url` key).
|
||||
|
||||
For example, if you pass ten URLs to `LinkContentFetcher` and it manages to fetch six of them, then the output will be a list of six `ByteStream` objects, each containing information about its content type and URL.
|
||||
|
||||
It may happen that some sites block `LinkContentFetcher` from getting their content. In that case, it logs the error and returns the `ByteStream` objects that it successfully fetched.
|
||||
|
||||
Often, to use this component in a pipeline, you must convert the returned list of `ByteStream` objects into a list of `Document` objects. To do so, you can use the `HTMLToDocument` component.
|
||||
|
||||
You can use `LinkContentFetcher` at the beginning of an indexing pipeline to index the contents of URLs into a Document Store. You can also use it directly in a query pipeline, such as a retrieval-augmented generative (RAG) pipeline, to use the contents of a URL as the data source.
|
||||
|
||||
## Security considerations
|
||||
|
||||
`LinkContentFetcher` requests the URLs passed to it. If those URLs come directly from end users, this can expose your environment to server-side request forgery (SSRF) risks.
|
||||
|
||||
Before calling `LinkContentFetcher`, an application should therefore validate and sanitize user-provided URLs. For example:
|
||||
|
||||
- Allow only expected schemes, for example `https`
|
||||
- Use an allowlist of trusted domains when possible
|
||||
- Block localhost, link-local, and private-network destinations
|
||||
- Consider using an outbound proxy or network-level egress restrictions in production
|
||||
|
||||
For example, an application could block private, loopback, link-local, reserved IPs, and custom IP ranges using the standard library's `ipaddress` module:
|
||||
|
||||
```python
|
||||
import ipaddress
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
PRIVATE_RANGES = (
|
||||
ipaddress.ip_network("127.0.0.0/8"),
|
||||
ipaddress.ip_network("10.0.0.0/8"),
|
||||
ipaddress.ip_network("172.16.0.0/12"),
|
||||
ipaddress.ip_network("192.168.0.0/16"),
|
||||
ipaddress.ip_network("169.254.0.0/16"),
|
||||
)
|
||||
|
||||
|
||||
def is_unsafe_url(url: str) -> bool:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme != "https" or not parsed.hostname:
|
||||
return True
|
||||
try:
|
||||
ip = ipaddress.ip_address(parsed.hostname)
|
||||
except ValueError:
|
||||
# Hostname (not a raw IP). Apply your own domain allowlist policy here. Filter out "LOCALHOST" etc.
|
||||
return False
|
||||
return (
|
||||
ip.is_private
|
||||
or ip.is_loopback
|
||||
or ip.is_link_local
|
||||
or ip.is_reserved
|
||||
or any(ip in net for net in PRIVATE_RANGES)
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where `LinkContentFetcher` fetches the contents of a URL. It initializes the component using the default settings. To change the default component settings, such as `retry_attempts`, check out the API reference [docs](/reference/fetchers-api).
|
||||
|
||||
```python
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
|
||||
fetcher = LinkContentFetcher()
|
||||
|
||||
fetcher.run(urls=["https://haystack.deepset.ai"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of an indexing pipeline that uses the `LinkContentFetcher` to index the contents of the specified URLs into an `InMemoryDocumentStore`. Notice how it uses the `HTMLToDocument` component to convert the list of `ByteStream` objects to `Document` objects.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
fetcher = LinkContentFetcher()
|
||||
converter = HTMLToDocument()
|
||||
writer = DocumentWriter(document_store=document_store)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(instance=fetcher, name="fetcher")
|
||||
indexing_pipeline.add_component(instance=converter, name="converter")
|
||||
indexing_pipeline.add_component(instance=writer, name="writer")
|
||||
|
||||
indexing_pipeline.connect("fetcher.streams", "converter.sources")
|
||||
indexing_pipeline.connect("converter.documents", "writer.documents")
|
||||
|
||||
indexing_pipeline.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://haystack.deepset.ai/blog/guide-to-using-zephyr-with-haystack2",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: "MSSharePointFetcher"
|
||||
id: mssharepointfetcher
|
||||
slug: "/mssharepointfetcher"
|
||||
description: "Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams."
|
||||
---
|
||||
|
||||
# MSSharePointFetcher
|
||||
|
||||
Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), before a Router or File Converters |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver` <br /> <br />`targets`: A list of `Document`s (from `MSSharePointRetriever`) or raw SharePoint/OneDrive `web_url` strings |
|
||||
| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content |
|
||||
| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint |
|
||||
| **Package name** | `microsoft-sharepoint-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MSSharePointFetcher` downloads the full content of Microsoft SharePoint and OneDrive items through the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/use-the-api) and returns `ByteStream` objects, ready for a downstream converter.
|
||||
|
||||
It complements [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), which returns only Search snippets and metadata. Wire the retriever's `documents` (or a list of `web_url`s) into the fetcher to download the underlying content. The fetcher dispatches on the entity type of each hit:
|
||||
|
||||
- **Files** (`driveItem`) are downloaded as their raw bytes (PDF, DOCX, ...).
|
||||
- **List items** (`listItem`) are returned as a JSON `ByteStream` of the item's column values (`fields`).
|
||||
- **SharePoint pages** (`sitePage`) are returned as an HTML `ByteStream` built from the page's web parts.
|
||||
|
||||
Each `ByteStream`'s `meta` carries `url`, `file_name`, `content_type`, and a normalized `entity_type` (`driveItem`, `listItem`, or `sitePage`). Everything is resolved through the Microsoft Graph `shares` endpoint (plus the Pages API for pages), so only the `web_url` already exposed by the retriever is needed.
|
||||
|
||||
Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`HTMLToDocument`](../converters/htmltodocument.mdx), or a JSON converter).
|
||||
|
||||
### Authentication
|
||||
|
||||
The fetcher takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All` for files and `Sites.Read.All` for list items and pages). Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.
|
||||
|
||||
### Error handling and concurrency
|
||||
|
||||
- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the item is skipped, so the remaining items are still returned.
|
||||
- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors.
|
||||
- `max_concurrent_requests` (default `5`): bounds the number of items fetched concurrently by `run_async` to avoid tripping Microsoft Graph rate limits. It has no effect on the synchronous `run`, which fetches items one at a time.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Microsoft SharePoint integration with:
|
||||
|
||||
```shell
|
||||
pip install microsoft-sharepoint-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
`access_token` below is a per-user delegated Microsoft Graph bearer token. You can pass either raw `web_url` strings or the `Document`s produced by `MSSharePointRetriever`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.fetchers.microsoft_sharepoint import (
|
||||
MSSharePointFetcher,
|
||||
)
|
||||
|
||||
fetcher = MSSharePointFetcher()
|
||||
|
||||
result = fetcher.run(
|
||||
access_token="my-delegated-graph-token",
|
||||
targets=[
|
||||
"https://contoso.sharepoint.com/sites/contoso-team/contoso-designs.docx",
|
||||
],
|
||||
)
|
||||
|
||||
for stream in result["streams"]:
|
||||
print(stream.meta["file_name"], stream.meta["content_type"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) searches SharePoint, `MSSharePointFetcher` downloads the matching items, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.routers import FileTypeRouter
|
||||
from haystack.components.converters import PyPDFToDocument, DOCXToDocument
|
||||
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
|
||||
MSSharePointRetriever,
|
||||
)
|
||||
from haystack_integrations.components.fetchers.microsoft_sharepoint import (
|
||||
MSSharePointFetcher,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"resolver",
|
||||
OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
|
||||
scopes=[
|
||||
"https://graph.microsoft.com/Files.Read.All",
|
||||
"https://graph.microsoft.com/Sites.Read.All",
|
||||
"offline_access",
|
||||
],
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
|
||||
pipeline.add_component("fetcher", MSSharePointFetcher())
|
||||
pipeline.add_component(
|
||||
"router",
|
||||
FileTypeRouter(
|
||||
mime_types=[
|
||||
"application/pdf",
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
],
|
||||
),
|
||||
)
|
||||
pipeline.add_component("pdf_converter", PyPDFToDocument())
|
||||
pipeline.add_component("docx_converter", DOCXToDocument())
|
||||
|
||||
# The same token feeds both the retriever and the fetcher.
|
||||
pipeline.connect("resolver.access_token", "retriever.access_token")
|
||||
pipeline.connect("resolver.access_token", "fetcher.access_token")
|
||||
|
||||
# The retrieved documents become the fetcher's targets.
|
||||
pipeline.connect("retriever.documents", "fetcher.targets")
|
||||
|
||||
# Route each fetched ByteStream to the matching converter.
|
||||
pipeline.connect("fetcher.streams", "router.sources")
|
||||
pipeline.connect("router.application/pdf", "pdf_converter.sources")
|
||||
pipeline.connect(
|
||||
"router.application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
"docx_converter.sources",
|
||||
)
|
||||
|
||||
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
|
||||
```
|
||||
Reference in New Issue
Block a user