chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "BraveWebSearch"
|
||||
id: bravewebsearch
|
||||
slug: "/bravewebsearch"
|
||||
description: "Search engine using the Brave Search API."
|
||||
---
|
||||
|
||||
# BraveWebSearch
|
||||
|
||||
Search the web using the Brave Search API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Brave Search API key. Can be set with the `BRAVE_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your search query. |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents containing search result content and metadata. <br /> <br />`links`: A list of strings of resulting URLs. |
|
||||
| **API reference** | [Brave Search API](/reference/integrations-brave) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/brave/src/haystack_integrations/components/websearch/brave/brave_websearch.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `BraveWebSearch` a query, it uses the [Brave Search API](https://brave.com/search/api/) to search the web and return relevant content as Haystack `Document` objects. It also returns a list of the source URLs.
|
||||
|
||||
Brave Search is an independent search engine with its own web index. It is a great fit for RAG pipelines that need reliable, privacy-focused web results without depending on Google or Bing.
|
||||
|
||||
`BraveWebSearch` requires a Brave Search API key to work. By default, it looks for a `BRAVE_API_KEY` environment variable. Alternatively, you can pass an `api_key` directly during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is a quick example of how `BraveWebSearch` searches the web based on a query and returns a list of Documents.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.websearch.brave import BraveWebSearch
|
||||
from haystack.utils import Secret
|
||||
|
||||
web_search = BraveWebSearch(
|
||||
api_key=Secret.from_env_var("BRAVE_API_KEY"),
|
||||
top_k=5,
|
||||
)
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
response = web_search.run(query=query)
|
||||
|
||||
for doc in response["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here is an example of a Retrieval-Augmented Generation (RAG) pipeline that uses `BraveWebSearch` to look up an answer on the web.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.websearch.brave import BraveWebSearch
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
web_search = BraveWebSearch(
|
||||
api_key=Secret.from_env_var("BRAVE_API_KEY"),
|
||||
top_k=3,
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}\n{% endfor %}\n"
|
||||
"Answer the following question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
|
||||
llm = OpenAIChatGenerator(
|
||||
api_key=Secret.from_env_var("OPENAI_API_KEY"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
result = pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-websearch
|
||||
slug: "/external-integrations-websearch"
|
||||
description: "External integrations that enable websearch with Haystack."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable websearch with Haystack.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [DuckDuckGo](https://haystack.deepset.ai/integrations/duckduckgo-api-websearch) | Use DuckDuckGo API for web searches. |
|
||||
| [Exa](https://haystack.deepset.ai/integrations/exa) | Search the web with Exa's AI-powered search, get content, answers, and conduct deep research. |
|
||||
| [Serpex](https://haystack.deepset.ai/integrations/serpex) | Multi-engine web search for Haystack — access Google, Bing, DuckDuckGo, Brave, Yahoo, and Yandex via Serpex API. |
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
title: "FirecrawlWebSearch"
|
||||
id: firecrawlwebsearch
|
||||
slug: "/firecrawlwebsearch"
|
||||
description: "Search engine using the Firecrawl API."
|
||||
---
|
||||
|
||||
# FirecrawlWebSearch
|
||||
|
||||
Search the web and extract content using the Firecrawl API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or right at the beginning of an indexing pipeline. |
|
||||
| **Mandatory init variables** | `api_key`: The Firecrawl API key. Can be set with the `FIRECRAWL_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your search query. |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents containing the scraped content and metadata. <br /> <br />`links`: A list of strings of resulting URLs. |
|
||||
| **API reference** | [Firecrawl Search API](/reference/integrations-firecrawl) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/firecrawl/src/haystack_integrations/components/websearch/firecrawl/firecrawl_websearch.py |
|
||||
| **Package name** | `firecrawl-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `FirecrawlWebSearch` a query, it uses the Firecrawl Search API to search the web, crawl the resulting pages, and return the structured text as a list of Haystack `Document` objects. It also returns a list of the underlying URLs.
|
||||
|
||||
Because Firecrawl actively scrapes and structures the content of the pages it finds into LLM-friendly formats, you generally don't need an additional component like `LinkContentFetcher` to read the web pages. `FirecrawlWebSearch` handles the retrieval and scraping all in one step.
|
||||
|
||||
`FirecrawlWebSearch` requires a [Firecrawl](https://firecrawl.dev) API key to work. By default, it looks for a `FIRECRAWL_API_KEY` environment variable. Alternatively, you can pass an `api_key` directly during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is a quick example of how `FirecrawlWebSearch` searches the web based on a query, scrapes the resulting web pages, and returns a list of Documents containing the page content.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.websearch.firecrawl import FirecrawlWebSearch
|
||||
from haystack.utils import Secret
|
||||
|
||||
web_search = FirecrawlWebSearch(
|
||||
api_key=Secret.from_env_var("FIRECRAWL_API_KEY"),
|
||||
top_k=5,
|
||||
search_params={"scrape_options": {"formats": ["markdown"]}},
|
||||
)
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
response = web_search.run(query=query)
|
||||
|
||||
for doc in response["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here is an example of a Retrieval-Augmented Generation (RAG) pipeline where using `FirecrawlWebSearch` to look up an answer. Because Firecrawl returns the actual text of the scraped pages, you can pass its `documents` output directly into the `ChatPromptBuilder` to give the LLM the necessary context.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.websearch.firecrawl import FirecrawlWebSearch
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
web_search = FirecrawlWebSearch(
|
||||
api_key=Secret.from_env_var("FIRECRAWL_API_KEY"),
|
||||
top_k=2,
|
||||
search_params={"scrape_options": {"formats": ["markdown"]}},
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}\n{% endfor %}\n"
|
||||
"Answer the following question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
|
||||
llm = OpenAIChatGenerator(
|
||||
api_key=Secret.from_env_var("OPENAI_API_KEY"),
|
||||
model="gpt-5-nano",
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
result = pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: "PerplexityWebSearch"
|
||||
id: perplexitywebsearch
|
||||
slug: "/perplexitywebsearch"
|
||||
description: "Search the web using the Perplexity Search API and return results as Haystack Documents."
|
||||
---
|
||||
|
||||
# PerplexityWebSearch
|
||||
|
||||
Search the web using the Perplexity Search API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: A Perplexity API key. Can be set with `PERPLEXITY_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your search query. |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents containing search result content and metadata. <br /> <br />`links`: A list of strings of resulting URLs. |
|
||||
| **API reference** | [Integrations](/reference/integrations-perplexity) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/perplexity/src/haystack_integrations/components/websearch/perplexity/perplexity_websearch.py |
|
||||
| **Package name** | `perplexity-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `PerplexityWebSearch` a query, it uses the [Perplexity Search API](https://docs.perplexity.ai/) to search the web and return relevant content as Haystack `Document` objects. It also returns a list of the source URLs.
|
||||
|
||||
Each returned `Document` contains a text snippet as its `content` and a `meta` dictionary with `title`, `url`, `date`, and `last_updated` fields.
|
||||
|
||||
`PerplexityWebSearch` requires a Perplexity API key to work. By default, it reads from the `PERPLEXITY_API_KEY` environment variable. You can also pass an `api_key` directly during initialization.
|
||||
|
||||
The `top_k` parameter controls the maximum number of results returned (between 1 and 20, default is 10).
|
||||
|
||||
You can filter and refine search results using `search_params`, which supports keys such as `country`, `search_recency_filter`, `search_domain_filter`, and date range filters. These can be set at initialization or overridden per `run()` call. See the [Perplexity Search API reference](https://docs.perplexity.ai/api-reference/search-post) for the full list of parameters.
|
||||
|
||||
`PerplexityWebSearch` supports both synchronous (`run()`) and asynchronous (`run_async()`) operation.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.websearch.perplexity import PerplexityWebSearch
|
||||
|
||||
web_search = PerplexityWebSearch(
|
||||
api_key=Secret.from_env_var("PERPLEXITY_API_KEY"),
|
||||
top_k=5,
|
||||
)
|
||||
result = web_search.run(query="What is Haystack by deepset?")
|
||||
|
||||
for doc in result["documents"]:
|
||||
print(doc.content)
|
||||
print(doc.meta["url"])
|
||||
```
|
||||
|
||||
With search filters:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.websearch.perplexity import PerplexityWebSearch
|
||||
|
||||
web_search = PerplexityWebSearch(
|
||||
api_key=Secret.from_env_var("PERPLEXITY_API_KEY"),
|
||||
top_k=5,
|
||||
search_params={"country": "us", "search_recency_filter": "week"},
|
||||
)
|
||||
result = web_search.run(query="Latest AI research papers")
|
||||
|
||||
for doc in result["documents"]:
|
||||
print(doc.meta["title"], doc.meta["url"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here is an example of a RAG pipeline that uses `PerplexityWebSearch` to look up an answer on the web.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.perplexity import (
|
||||
PerplexityChatGenerator,
|
||||
)
|
||||
from haystack_integrations.components.websearch.perplexity import PerplexityWebSearch
|
||||
|
||||
web_search = PerplexityWebSearch(
|
||||
api_key=Secret.from_env_var("PERPLEXITY_API_KEY"),
|
||||
top_k=3,
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}\n{% endfor %}\n"
|
||||
"Answer the following question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables=["query", "documents"],
|
||||
)
|
||||
|
||||
llm = PerplexityChatGenerator(
|
||||
api_key=Secret.from_env_var("PERPLEXITY_API_KEY"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is Haystack by deepset?"
|
||||
result = pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "SearchApiWebSearch"
|
||||
id: searchapiwebsearch
|
||||
slug: "/searchapiwebsearch"
|
||||
description: "Search engine using Search API."
|
||||
---
|
||||
|
||||
# SearchApiWebSearch
|
||||
|
||||
Search engine using Search API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) or [Converters](../converters.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The SearchAPI API key. Can be set with `SEARCHAPI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your query |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`links`: A list of strings of resulting links |
|
||||
| **API reference** | [SearchApi](/reference/integrations-searchapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/searchapi |
|
||||
| **Package name** | `searchapi-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `SearchApiWebSearch` a query, it returns a list of the URLs most relevant to your search. It uses page snippets (pieces of text displayed under the page title in search results) to find the answers, not the whole pages.
|
||||
|
||||
To search the content of the web pages, use the [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) component.
|
||||
|
||||
`SearchApiWebSearch` requires a [SearchApi](https://www.searchapi.io) key to work. It uses a `SEARCHAPI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below.
|
||||
|
||||
:::info[Alternative search]
|
||||
|
||||
To use [Serper Dev](https://serper.dev/?gclid=Cj0KCQiAgqGrBhDtARIsAM5s0_kPElllv3M59UPok1Ad-ZNudLaY21zDvbt5qw-b78OcUoqqvplVHRwaAgRgEALw_wcB) as an alternative, see its respective [documentation page](serperdevwebsearch.mdx).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `searchapi-haystack` package to use the `SearchApiWebSearch` component:
|
||||
|
||||
```shell
|
||||
pip install searchapi-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This is an example of how `SearchApiWebSearch` looks up answers to our query on the web and converts the results into a list of documents with content snippets of the results, as well as URLs as strings.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.websearch.searchapi import SearchApiWebSearch
|
||||
|
||||
web_search = SearchApiWebSearch(api_key=Secret.from_token("<your-api-key>"))
|
||||
query = "What is the capital of Germany?"
|
||||
|
||||
response = web_search.run(query)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here’s an example of a RAG pipeline where we use a `SearchApiWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.websearch.searchapi import SearchApiWebSearch
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
web_search = SearchApiWebSearch(api_key=Secret.from_token("<your-api-key>"), top_k=2)
|
||||
link_content = LinkContentFetcher()
|
||||
html_converter = HTMLToDocument()
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}{% endfor %}\n"
|
||||
"Answer question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
llm = OpenAIChatGenerator(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("fetcher", link_content)
|
||||
pipe.add_component("converter", html_converter)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.links", "fetcher.urls")
|
||||
pipe.connect("fetcher.streams", "converter.sources")
|
||||
pipe.connect("converter.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is the most famous landmark in Berlin?"
|
||||
|
||||
pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
```
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: "SerperDevWebSearch"
|
||||
id: serperdevwebsearch
|
||||
slug: "/serperdevwebsearch"
|
||||
description: "Search engine using SerperDev API."
|
||||
---
|
||||
|
||||
# SerperDevWebSearch
|
||||
|
||||
Search engine using SerperDev API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) or [Converters](../converters.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The SearchAPI API key. Can be set with `SERPERDEV_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your query |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`links`: A list of strings of resulting links |
|
||||
| **API reference** | [SerperDev](/reference/integrations-serperdev) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/serperdev |
|
||||
| **Package name** | `serperdev-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `SerperDevWebSearch` a query, it returns a list of the URLs most relevant to your search. It uses page snippets (pieces of text displayed under the page title in search results) to find the answers, not the whole pages.
|
||||
|
||||
To search the content of the web pages, use the [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) component.
|
||||
|
||||
`SerperDevWebSearch` requires a [SerperDev](https://serper.dev/) key to work. It uses a `SERPERDEV_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below.
|
||||
|
||||
:::info[Alternative search]
|
||||
|
||||
To use [Search API](https://www.searchapi.io/) as an alternative, see its respective [documentation page](searchapiwebsearch.mdx).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `serperdev-haystack` package to use the `SerperDevWebSearch` component:
|
||||
|
||||
```shell
|
||||
pip install serperdev-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This is an example of how `SerperDevWebSearch` looks up answers to our query on the web and converts the results into a list of documents with content snippets of the results, as well as URLs as strings.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.websearch.serperdev import SerperDevWebSearch
|
||||
from haystack.utils import Secret
|
||||
|
||||
web_search = SerperDevWebSearch(api_key=Secret.from_token("<your-api-key>"))
|
||||
query = "What is the capital of Germany?"
|
||||
|
||||
response = web_search.run(query)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here’s an example of a RAG pipeline where we use a `SerperDevWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.websearch.serperdev import SerperDevWebSearch
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
web_search = SerperDevWebSearch(api_key=Secret.from_token("<your-api-key>"), top_k=2)
|
||||
link_content = LinkContentFetcher()
|
||||
html_converter = HTMLToDocument()
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}{% endfor %}\n"
|
||||
"Answer question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
llm = OpenAIChatGenerator(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("fetcher", link_content)
|
||||
pipe.add_component("converter", html_converter)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.links", "fetcher.urls")
|
||||
pipe.connect("fetcher.streams", "converter.sources")
|
||||
pipe.connect("converter.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is the most famous landmark in Berlin?"
|
||||
|
||||
pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
```
|
||||
|
||||
### In YAML
|
||||
This is the YAML representation of the RAG pipeline shown above. It searches the web, fetches the resulting pages, converts them to text, builds a prompt with the content, and generates an answer using a chat model.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
converter:
|
||||
init_parameters:
|
||||
extraction_kwargs: {}
|
||||
store_full_path: false
|
||||
type: haystack.components.converters.html.HTMLToDocument
|
||||
fetcher:
|
||||
init_parameters:
|
||||
client_kwargs:
|
||||
follow_redirects: true
|
||||
timeout: 3
|
||||
http2: false
|
||||
raise_on_failure: true
|
||||
request_headers: {}
|
||||
retry_attempts: 2
|
||||
timeout: 3
|
||||
user_agents:
|
||||
- haystack/LinkContentFetcher/2.27.0rc0
|
||||
type: haystack.components.fetchers.link_content.LinkContentFetcher
|
||||
llm:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-4o-mini
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables:
|
||||
- documents
|
||||
- query
|
||||
template:
|
||||
- content:
|
||||
- text: You are a helpful assistant.
|
||||
meta: {}
|
||||
name: null
|
||||
role: system
|
||||
- content:
|
||||
- text: 'Given the information below:
|
||||
|
||||
{% for document in documents %}{{ document.content }}{% endfor %}
|
||||
|
||||
Answer question: {{ query }}.
|
||||
|
||||
Answer:'
|
||||
meta: {}
|
||||
name: null
|
||||
role: user
|
||||
variables: null
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
search:
|
||||
init_parameters:
|
||||
allowed_domains: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- SERPERDEV_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
exclude_subdomains: false
|
||||
search_params: {}
|
||||
top_k: 2
|
||||
type: haystack_integrations.components.websearch.serperdev.websearch.SerperDevWebSearch
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: fetcher.urls
|
||||
sender: search.links
|
||||
- receiver: converter.sources
|
||||
sender: fetcher.streams
|
||||
- receiver: prompt_builder.documents
|
||||
sender: converter.documents
|
||||
- receiver: llm.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Building Fallbacks to Websearch with Conditional Routing](https://haystack.deepset.ai/tutorials/36_building_fallbacks_with_conditional_routing)
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: "TavilyWebSearch"
|
||||
id: tavilywebsearch
|
||||
slug: "/tavilywebsearch"
|
||||
description: "Search engine using the Tavily AI-powered search API."
|
||||
---
|
||||
|
||||
# TavilyWebSearch
|
||||
|
||||
Search the web using the Tavily AI-powered search API, optimized for LLM applications.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Tavily API key. Can be set with the `TAVILY_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A string with your search query. |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents containing search result content and metadata. <br /> <br />`links`: A list of strings of resulting URLs. |
|
||||
| **API reference** | [Tavily Search API](/reference/integrations-tavily) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/tavily/src/haystack_integrations/components/websearch/tavily/tavily_websearch.py |
|
||||
| **Package name** | `tavily-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
When you give `TavilyWebSearch` a query, it uses the [Tavily](https://tavily.com) Search API to search the web and return relevant content as Haystack `Document` objects. It also returns a list of the source URLs.
|
||||
|
||||
Tavily is an AI-powered search API built specifically for LLM applications. It returns clean, relevant snippets without the noise of traditional search engines, making it a great fit for RAG pipelines.
|
||||
|
||||
`TavilyWebSearch` requires a Tavily API key to work. By default, it looks for a `TAVILY_API_KEY` environment variable. Alternatively, you can pass an `api_key` directly during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is a quick example of how `TavilyWebSearch` searches the web based on a query and returns a list of Documents.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.websearch.tavily import TavilyWebSearch
|
||||
from haystack.utils import Secret
|
||||
|
||||
web_search = TavilyWebSearch(
|
||||
api_key=Secret.from_env_var("TAVILY_API_KEY"),
|
||||
top_k=5,
|
||||
)
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
response = web_search.run(query=query)
|
||||
|
||||
for doc in response["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here is an example of a Retrieval-Augmented Generation (RAG) pipeline that uses `TavilyWebSearch` to look up an answer on the web.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.websearch.tavily import TavilyWebSearch
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
web_search = TavilyWebSearch(
|
||||
api_key=Secret.from_env_var("TAVILY_API_KEY"),
|
||||
top_k=3,
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}\n{% endfor %}\n"
|
||||
"Answer the following question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
|
||||
llm = OpenAIChatGenerator(
|
||||
api_key=Secret.from_env_var("OPENAI_API_KEY"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("search", web_search)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("search.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is Haystack by deepset?"
|
||||
|
||||
result = pipe.run(data={"search": {"query": query}, "prompt_builder": {"query": query}})
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
Reference in New Issue
Block a user