chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "AnswerJoiner"
|
||||
id: answerjoiner
|
||||
slug: "/answerjoiner"
|
||||
description: "Merges multiple answers from different Generators into a single list."
|
||||
---
|
||||
|
||||
# AnswerJoiner
|
||||
|
||||
Merges multiple answers from different Generators into a single list.
|
||||
|
||||
| | |
|
||||
| :------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Most common position in a pipeline** | In query pipelines, after [Generators](../generators.mdx) and, subsequently, components that return a list of answers such as [`AnswerBuilder`](../builders/answerbuilder.mdx) |
|
||||
| **Mandatory run variables** | “answers”: A nested list of answers to be merged, received from the Generator. This input is `variadic`, meaning you can connect a variable number of components to it. |
|
||||
| **Output variables** | “answers”: A merged list of answers |
|
||||
| **API reference** | [Joiners](/reference/joiners-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/joiners/answer_joiner.py |
|
||||
|
||||
## Overvew
|
||||
|
||||
`AnswerJoiner` joins input lists of [`Answer`](../../concepts/data-classes.mdx#answer) objects from multiple connections and returns them as one list.
|
||||
|
||||
You can optionally set the `top_k` parameter, which specifies the maximum number of answers to return. If you don’t set this parameter, the component returns all answers it receives.
|
||||
|
||||
## Usage
|
||||
|
||||
In this simple example pipeline, the `AnswerJoiner` merges answers from two instances of Generators:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import AnswerBuilder
|
||||
from haystack.components.joiners import AnswerJoiner
|
||||
|
||||
from haystack.core.pipeline import Pipeline
|
||||
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
query = "What's Natural Language Processing?"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful, respectful and honest assistant. Be super concise.",
|
||||
),
|
||||
ChatMessage.from_user(query),
|
||||
]
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("gpt-4o", OpenAIChatGenerator(model="gpt-4o"))
|
||||
pipe.add_component("llama", OpenAIChatGenerator(model="gpt-3.5-turbo"))
|
||||
pipe.add_component("aba", AnswerBuilder())
|
||||
pipe.add_component("abb", AnswerBuilder())
|
||||
pipe.add_component("joiner", AnswerJoiner())
|
||||
|
||||
pipe.connect("gpt-4o.replies", "aba")
|
||||
pipe.connect("llama.replies", "abb")
|
||||
pipe.connect("aba.answers", "joiner")
|
||||
pipe.connect("abb.answers", "joiner")
|
||||
|
||||
results = pipe.run(
|
||||
data={
|
||||
"gpt-4o": {"messages": messages},
|
||||
"llama": {"messages": messages},
|
||||
"aba": {"query": query},
|
||||
"abb": {"query": query},
|
||||
},
|
||||
)
|
||||
```
|
||||
+223
@@ -0,0 +1,223 @@
|
||||
---
|
||||
title: "BranchJoiner"
|
||||
id: branchjoiner
|
||||
slug: "/branchjoiner"
|
||||
description: "Use this component to join different branches of a pipeline into a single output."
|
||||
---
|
||||
|
||||
import ClickableImage from "@site/src/components/ClickableImage";
|
||||
|
||||
# BranchJoiner
|
||||
|
||||
Use this component to join different branches of a pipeline into a single output.
|
||||
|
||||
| | |
|
||||
| :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Most common position in a pipeline** | Flexible: Can appear at the beginning of a pipeline or at the start of loops. |
|
||||
| **Mandatory init variables** | "type": The type of data expected from preceding components |
|
||||
| **Mandatory run variables** | “\*\*kwargs”: Any input data type defined at the initialization. This input is variadic, meaning you can connect a variable number of components to it. |
|
||||
| **Output variables** | “value”: The first value received from the connected components. |
|
||||
| **API reference** | [Joiners](/reference/joiners-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/joiners/branch.py |
|
||||
|
||||
## Overview
|
||||
|
||||
`BranchJoiner` joins multiple branches in a pipeline, allowing their outputs to be reconciled into a single branch. This is especially useful in pipelines with multiple branches that need to be unified before moving to the single component that comes next.
|
||||
|
||||
`BranchJoiner` receives multiple data connections of the same type from other components and passes the first value it receives to its single output. This makes it essential for closing loops in pipelines or reconciling multiple branches from a decision component.
|
||||
|
||||
`BranchJoiner` can handle only one input of one data type, declared in the `__init__` function. It ensures that the data type remains consistent across the pipeline branches. If more than one value is received for the input when `run` is invoked, the component will raise an error:
|
||||
|
||||
```python
|
||||
from haystack.components.joiners import BranchJoiner
|
||||
|
||||
bj = BranchJoiner(int)
|
||||
bj.run(value=[3, 4, 5])
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Although only one input value is allowed at every run, due to its variadic nature `BranchJoiner` still expects a list. As an example:
|
||||
|
||||
```python
|
||||
from haystack.components.joiners import BranchJoiner
|
||||
|
||||
## an example where input and output are strings
|
||||
bj = BranchJoiner(str)
|
||||
bj.run(value=["hello"])
|
||||
|
||||
## an example where input and output are integers
|
||||
bj = BranchJoiner(int)
|
||||
bj.run(value=[3])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
#### Enabling loops
|
||||
|
||||
Below is an example where `BranchJoiner` is used for closing a loop. In this example, `BranchJoiner` receives a looped-back list of `ChatMessage` objects from the `JsonSchemaValidator` and sends it down to the `OpenAIChatGenerator` for re-generation.
|
||||
|
||||
```python
|
||||
import json
|
||||
from typing import List
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.joiners import BranchJoiner
|
||||
from haystack.components.validators import JsonSchemaValidator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
person_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"first_name": {"type": "string", "pattern": "^[A-Z][a-z]+$"},
|
||||
"last_name": {"type": "string", "pattern": "^[A-Z][a-z]+$"},
|
||||
"nationality": {
|
||||
"type": "string",
|
||||
"enum": ["Italian", "Portuguese", "American"],
|
||||
},
|
||||
},
|
||||
"required": ["first_name", "last_name", "nationality"],
|
||||
}
|
||||
|
||||
## Initialize a pipeline
|
||||
pipe = Pipeline()
|
||||
|
||||
## Add components to the pipeline
|
||||
pipe.add_component("joiner", BranchJoiner(List[ChatMessage]))
|
||||
pipe.add_component("fc_llm", OpenAIChatGenerator(model="gpt-4o-mini"))
|
||||
pipe.add_component("validator", JsonSchemaValidator(json_schema=person_schema))
|
||||
pipe.add_component(
|
||||
"adapter",
|
||||
OutputAdapter("{{chat_message}}", List[ChatMessage], unsafe=True),
|
||||
)
|
||||
|
||||
## Connect components
|
||||
pipe.connect("adapter", "joiner")
|
||||
pipe.connect("joiner", "fc_llm")
|
||||
pipe.connect("fc_llm.replies", "validator.messages")
|
||||
pipe.connect("validator.validation_error", "joiner")
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fc_llm": {"generation_kwargs": {"response_format": {"type": "json_object"}}},
|
||||
"adapter": {
|
||||
"chat_message": [
|
||||
ChatMessage.from_user("Create json object from Peter Parker"),
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(json.loads(result["validator"]["validated"][0].text))
|
||||
|
||||
## Output:
|
||||
## {'first_name': 'Peter', 'last_name': 'Parker', 'nationality': 'American', 'name': 'Spider-Man', 'occupation':
|
||||
## 'Superhero', 'age': 23, 'location': 'New York City'}
|
||||
```
|
||||
|
||||
<details>
|
||||
|
||||
<summary>Expand to see the pipeline graph</summary>
|
||||
|
||||
<ClickableImage src="/img/9dc767d-loop_chart.png" alt="Pipeline flowchart showing a validation loop with adapter, joiner, language model, and validator components forming a cycle until validation succeeds" size="large" />
|
||||
|
||||
</details>
|
||||
|
||||
#### Reconciling branches
|
||||
|
||||
In this example, the `TextLanguageRouter` component directs the query to one of three language-specific Retrievers. The next component would be a `PromptBuilder`, but we cannot connect multiple Retrievers to a single `PromptBuilder` directly. Instead, we connect all the Retrievers to the `BranchJoiner` component. The `BranchJoiner` then takes the output from the Retriever that was actually called and passes it as a single list of documents to the `PromptBuilder`. The `BranchJoiner` ensures that the pipeline can handle multiple languages seamlessly by consolidating different outputs from the Retrievers into a unified connection for further processing.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.joiners import BranchJoiner
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack.components.generators import OpenAIGenerator
|
||||
from haystack.components.routers import TextLanguageRouter
|
||||
|
||||
prompt_template = """
|
||||
Answer the question based on the given reviews.
|
||||
Reviews:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
Question: {{ query}}
|
||||
Answer:
|
||||
"""
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Super appartement. Juste au dessus de plusieurs bars qui ferment très tard. A savoir à l'avance. (Bouchons d'oreilles fournis !)",
|
||||
),
|
||||
Document(
|
||||
content="El apartamento estaba genial y muy céntrico, todo a mano. Al lado de la librería Lello y De la Torre de los clérigos. Está situado en una zona de marcha, así que si vais en fin de semana , habrá ruido, aunque a nosotros no nos molestaba para dormir",
|
||||
),
|
||||
Document(
|
||||
content="The keypad with a code is convenient and the location is convenient. Basically everything else, very noisy, wi-fi didn't work, check-in person didn't explain anything about facilities, shower head was broken, there's no cleaning and everything else one may need is charged.",
|
||||
),
|
||||
Document(
|
||||
content="It is very central and appartement has a nice appearance (even though a lot IKEA stuff), *W A R N I N G** the appartement presents itself as a elegant and as a place to relax, very wrong place to relax - you cannot sleep in this appartement, even the beds are vibrating from the bass of the clubs in the same building - you get ear plugs from the hotel.",
|
||||
),
|
||||
Document(
|
||||
content="Céntrico. Muy cómodo para moverse y ver Oporto. Edificio con terraza propia en la última planta. Todo reformado y nuevo. They traen un estupendo desayuno todas las mañanas al apartamento. Solo que se puede escuchar algo de ruido de la called a primeras horas de la noche. Es un zona de ocio nocturno. Pero respetan los horarios.",
|
||||
),
|
||||
]
|
||||
|
||||
en_document_store = InMemoryDocumentStore()
|
||||
fr_document_store = InMemoryDocumentStore()
|
||||
es_document_store = InMemoryDocumentStore()
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(
|
||||
instance=TextLanguageRouter(["en", "fr", "es"]),
|
||||
name="router",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=en_document_store),
|
||||
name="en_retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=fr_document_store),
|
||||
name="fr_retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=es_document_store),
|
||||
name="es_retriever",
|
||||
)
|
||||
rag_pipeline.add_component(instance=BranchJoiner(type_=list[Document]), name="joiner")
|
||||
rag_pipeline.add_component(
|
||||
instance=PromptBuilder(template=prompt_template),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIGenerator(), name="llm")
|
||||
|
||||
rag_pipeline.connect("router.en", "en_retriever.query")
|
||||
rag_pipeline.connect("router.fr", "fr_retriever.query")
|
||||
rag_pipeline.connect("router.es", "es_retriever.query")
|
||||
rag_pipeline.connect("en_retriever", "joiner")
|
||||
rag_pipeline.connect("fr_retriever", "joiner")
|
||||
rag_pipeline.connect("es_retriever", "joiner")
|
||||
rag_pipeline.connect("joiner", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder", "llm")
|
||||
|
||||
en_question = "Does this apartment has a noise problem?"
|
||||
|
||||
result = rag_pipeline.run(
|
||||
{"router": {"text": en_question}, "prompt_builder": {"query": en_question}},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0])
|
||||
```
|
||||
|
||||
<details>
|
||||
|
||||
<summary>Expand to see the pipeline graph</summary>
|
||||
|
||||
<ClickableImage src="/img/6da5ddd-join_chart.png" alt="Pipeline flowchart demonstrating BranchJoiner reconciling outputs from three language-specific retrievers into a single prompt builder" />
|
||||
|
||||
</details>
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
---
|
||||
title: "DocumentJoiner"
|
||||
id: documentjoiner
|
||||
slug: "/documentjoiner"
|
||||
description: "Use this component in hybrid retrieval pipelines or indexing pipelines with multiple file converters to join lists of documents."
|
||||
---
|
||||
|
||||
# DocumentJoiner
|
||||
|
||||
Use this component in hybrid retrieval pipelines or indexing pipelines with multiple file converters to join lists of documents.
|
||||
|
||||
| | |
|
||||
| :------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Most common position in a pipeline** | In indexing and query pipelines, after components that return a list of documents such as multiple [Retrievers](../retrievers.mdx) or multiple [Converters](../converters.mdx) |
|
||||
| **Mandatory run variables** | “documents”: A list of documents. This input is `variadic`, meaning you can connect a variable number of components to it. |
|
||||
| **Output variables** | “documents”: A list of documents |
|
||||
| **API reference** | [Joiners](/reference/joiners-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/joiners/document_joiner.py |
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentJoiner` joins input lists of documents from multiple connections and outputs them as one list. You can choose how you want the lists to be joined by specifying the `join_mode`. There are three options available:
|
||||
|
||||
- `concatenate` - Combines document from multiple components, discarding any duplicates. documents get their scores from the last component in the pipeline that assigns scores. This mode doesn’t influence document scores.
|
||||
- `merge` - Merges the scores of duplicate documents coming from multiple components. You can also assign a weight to the scores to influence how they’re merged and set the top_k limit to specify how many documents you want `DocumentJoiner` to return.
|
||||
- `reciprocal_rank_fusion`- Combines documents into a single list based on their ranking received from multiple components. It then calculates a new score based on the ranks of documents in the input lists. If the same Document appears in more than one list (was returned by multiple components), it gets a higher score.
|
||||
- `distribution_based_rank_fusion` – Combines rankings from multiple sources into a single, unified ranking. It analyzes how scores are spread out and normalizes them, ensuring that each component's scoring method is taken into account. This normalization helps to balance the influence of each component, resulting in a more robust and fair combined ranking. If a document appears in multiple lists, its final score is adjusted based on the distribution of scores from all lists.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we are using the `DocumentJoiner` to merge two lists of documents. We run the `DocumentJoiner` and provide the documents. It returns a list of documents ranked by combined scores. By default, equal weight is given to each Retriever score. You could also use custom weights by setting the weights parameter to a list of floats with one weight per input component.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.joiners.document_joiner import DocumentJoiner
|
||||
|
||||
docs_1 = [
|
||||
Document(content="Paris is the capital of France.", score=0.5),
|
||||
Document(content="Berlin is the capital of Germany.", score=0.4),
|
||||
]
|
||||
docs_2 = [
|
||||
Document(content="Paris is the capital of France.", score=0.6),
|
||||
Document(content="Rome is the capital of Italy.", score=0.5),
|
||||
]
|
||||
|
||||
joiner = DocumentJoiner(join_mode="merge")
|
||||
|
||||
joiner.run(documents=[docs_1, docs_2])
|
||||
|
||||
## {'documents': [Document(id=0f5beda04153dbfc462c8b31f8536749e43654709ecf0cfe22c6d009c9912214, content: 'Paris is the capital of France.', score: 0.55), Document(id=424beed8b549a359239ab000f33ca3b1ddb0f30a988bbef2a46597b9c27e42f2, content: 'Rome is the capital of Italy.', score: 0.25), Document(id=312b465e77e25c11512ee76ae699ce2eb201f34c8c51384003bb367e24fb6cf8, content: 'Berlin is the capital of Germany.', score: 0.2)]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
#### Hybrid Retrieval
|
||||
|
||||
Below is an example of a hybrid retrieval pipeline that retrieves documents from an `InMemoryDocumentStore` based on keyword search (using `InMemoryBM25Retriever`) and embedding search (using `InMemoryEmbeddingRetriever`). It then uses the `DocumentJoiner` with its default join mode to concatenate the retrieved documents into one list. The Document Store must contain documents with embeddings, otherwise the `InMemoryEmbeddingRetriever` will not return any documents.
|
||||
|
||||
```python
|
||||
from haystack.components.joiners.document_joiner import DocumentJoiner
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers.in_memory import (
|
||||
InMemoryBM25Retriever,
|
||||
InMemoryEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.embedders import SentenceTransformersTextEmbedder
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=document_store),
|
||||
name="bm25_retriever",
|
||||
)
|
||||
p.add_component(
|
||||
instance=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
name="text_embedder",
|
||||
)
|
||||
p.add_component(
|
||||
instance=InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
name="embedding_retriever",
|
||||
)
|
||||
p.add_component(instance=DocumentJoiner(), name="joiner")
|
||||
p.connect("bm25_retriever", "joiner")
|
||||
p.connect("embedding_retriever", "joiner")
|
||||
p.connect("text_embedder", "embedding_retriever")
|
||||
query = "What is the capital of France?"
|
||||
p.run(data={"bm25_retriever": {"query": query}, "text_embedder": {"text": query}})
|
||||
```
|
||||
|
||||
#### Indexing
|
||||
|
||||
Here's an example of an indexing pipeline that uses `DocumentJoiner` to compile all files into a single list of documents that can be fed through the rest of the indexing pipeline as one.
|
||||
|
||||
```python
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.converters import (
|
||||
MarkdownToDocument,
|
||||
PyPDFToDocument,
|
||||
TextFileToDocument,
|
||||
)
|
||||
from haystack.components.preprocessors import DocumentSplitter, DocumentCleaner
|
||||
from haystack.components.routers import FileTypeRouter
|
||||
from haystack.components.joiners import DocumentJoiner
|
||||
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from pathlib import Path
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
file_type_router = FileTypeRouter(
|
||||
mime_types=["text/plain", "application/pdf", "text/markdown"],
|
||||
)
|
||||
text_file_converter = TextFileToDocument()
|
||||
markdown_converter = MarkdownToDocument()
|
||||
pdf_converter = PyPDFToDocument()
|
||||
document_joiner = DocumentJoiner()
|
||||
|
||||
document_cleaner = DocumentCleaner()
|
||||
document_splitter = DocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=150,
|
||||
split_overlap=50,
|
||||
)
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
document_writer = DocumentWriter(document_store)
|
||||
|
||||
preprocessing_pipeline = Pipeline()
|
||||
preprocessing_pipeline.add_component(instance=file_type_router, name="file_type_router")
|
||||
preprocessing_pipeline.add_component(
|
||||
instance=text_file_converter,
|
||||
name="text_file_converter",
|
||||
)
|
||||
preprocessing_pipeline.add_component(
|
||||
instance=markdown_converter,
|
||||
name="markdown_converter",
|
||||
)
|
||||
preprocessing_pipeline.add_component(instance=pdf_converter, name="pypdf_converter")
|
||||
preprocessing_pipeline.add_component(instance=document_joiner, name="document_joiner")
|
||||
preprocessing_pipeline.add_component(instance=document_cleaner, name="document_cleaner")
|
||||
preprocessing_pipeline.add_component(
|
||||
instance=document_splitter,
|
||||
name="document_splitter",
|
||||
)
|
||||
preprocessing_pipeline.add_component(
|
||||
instance=document_embedder,
|
||||
name="document_embedder",
|
||||
)
|
||||
preprocessing_pipeline.add_component(instance=document_writer, name="document_writer")
|
||||
|
||||
preprocessing_pipeline.connect(
|
||||
"file_type_router.text/plain",
|
||||
"text_file_converter.sources",
|
||||
)
|
||||
preprocessing_pipeline.connect(
|
||||
"file_type_router.application/pdf",
|
||||
"pypdf_converter.sources",
|
||||
)
|
||||
preprocessing_pipeline.connect(
|
||||
"file_type_router.text/markdown",
|
||||
"markdown_converter.sources",
|
||||
)
|
||||
preprocessing_pipeline.connect("text_file_converter", "document_joiner")
|
||||
preprocessing_pipeline.connect("pypdf_converter", "document_joiner")
|
||||
preprocessing_pipeline.connect("markdown_converter", "document_joiner")
|
||||
preprocessing_pipeline.connect("document_joiner", "document_cleaner")
|
||||
preprocessing_pipeline.connect("document_cleaner", "document_splitter")
|
||||
preprocessing_pipeline.connect("document_splitter", "document_embedder")
|
||||
preprocessing_pipeline.connect("document_embedder", "document_writer")
|
||||
|
||||
preprocessing_pipeline.run(
|
||||
{"file_type_router": {"sources": list(Path(output_dir).glob("**/*"))}},
|
||||
)
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline)
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "ListJoiner"
|
||||
id: listjoiner
|
||||
slug: "/listjoiner"
|
||||
description: "A component that joins multiple lists into a single flat list."
|
||||
---
|
||||
|
||||
# ListJoiner
|
||||
|
||||
A component that joins multiple lists into a single flat list.
|
||||
|
||||
| | |
|
||||
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Most common position in a pipeline** | In indexing and query pipelines, after components that return lists of documents such as multiple [Retrievers](../retrievers.mdx) or multiple [Converters](../converters.mdx) |
|
||||
| **Mandatory run variables** | “values”: The dictionary of lists to be joined |
|
||||
| **Output variables** | “values”: A dictionary with a `values` key containing the joined list |
|
||||
| **API reference** | [Joiners](/reference/joiners-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/joiners/list_joiner.py |
|
||||
|
||||
## Overview
|
||||
|
||||
The `ListJoiner` component combines multiple lists into one list. It is useful for combining multiple lists from different pipeline components, merging LLM responses, handling multi-step data processing, and gathering data from different sources into one list.
|
||||
|
||||
The items stay in order based on when each input list was processed in a pipeline.
|
||||
|
||||
You can optionally specify a `list_type_` parameter to set the expected type of the lists being joined (for example, `List[ChatMessage]`). If not set, `ListJoiner` will accept lists containing mixed data types.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.joiners import ListJoiner
|
||||
|
||||
list1 = ["Hello", "world"]
|
||||
list2 = ["This", "is", "Haystack"]
|
||||
list3 = ["ListJoiner", "Example"]
|
||||
|
||||
joiner = ListJoiner()
|
||||
|
||||
result = joiner.run(values=[list1, list2, list3])
|
||||
|
||||
print(result["values"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.components.joiners import ListJoiner
|
||||
from typing import List
|
||||
|
||||
user_message = [
|
||||
ChatMessage.from_user("Give a brief answer the following question: {{query}}"),
|
||||
]
|
||||
|
||||
feedback_prompt = """
|
||||
You are given a question and an answer.
|
||||
Your task is to provide a score and a brief feedback on the answer.
|
||||
Question: {{query}}
|
||||
Answer: {{response}}
|
||||
"""
|
||||
feedback_message = [ChatMessage.from_system(feedback_prompt)]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(template=user_message)
|
||||
feedback_prompt_builder = ChatPromptBuilder(template=feedback_message)
|
||||
llm = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
feedback_llm = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.add_component("feedback_prompt_builder", feedback_prompt_builder)
|
||||
pipe.add_component("feedback_llm", feedback_llm)
|
||||
pipe.add_component("list_joiner", ListJoiner(List[ChatMessage]))
|
||||
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
pipe.connect("prompt_builder.prompt", "list_joiner")
|
||||
pipe.connect("llm.replies", "list_joiner")
|
||||
pipe.connect("llm.replies", "feedback_prompt_builder.response")
|
||||
pipe.connect("feedback_prompt_builder.prompt", "feedback_llm.messages")
|
||||
pipe.connect("feedback_llm.replies", "list_joiner")
|
||||
|
||||
query = "What is nuclear physics?"
|
||||
ans = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {"template_variables": {"query": query}},
|
||||
"feedback_prompt_builder": {"template_variables": {"query": query}},
|
||||
},
|
||||
)
|
||||
|
||||
print(ans["list_joiner"]["values"])
|
||||
```
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
title: "StringJoiner"
|
||||
id: stringjoiner
|
||||
slug: "/stringjoiner"
|
||||
description: "Component to join strings from different components into a list of strings."
|
||||
---
|
||||
|
||||
# StringJoiner
|
||||
|
||||
Component to join strings from different components into a list of strings.
|
||||
|
||||
| | |
|
||||
| :------------------------------------- | :---------------------------------------------------------------------------------------------- |
|
||||
| **Most common position in a pipeline** | After at least two other components to join their strings |
|
||||
| **Mandatory run variables** | “strings”: Multiple strings from connected components. |
|
||||
| **Output variables** | “strings”: A list of merged strings |
|
||||
| **API reference** | [Joiners](/reference/joiners-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/joiners/string_joiner.py |
|
||||
|
||||
## Overview
|
||||
|
||||
The `StringJoiner` component collects multiple string outputs from various pipeline components and combines them into a single list. This is useful when you need to merge several strings from different parts of a pipeline into a unified output.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from haystack.components.joiners import StringJoiner
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack.core.pipeline import Pipeline
|
||||
|
||||
string_1 = "What's Natural Language Processing?"
|
||||
string_2 = "What is life?"
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("prompt_builder_1", PromptBuilder("Builder 1: {{query}}"))
|
||||
pipeline.add_component("prompt_builder_2", PromptBuilder("Builder 2: {{query}}"))
|
||||
pipeline.add_component("string_joiner", StringJoiner())
|
||||
|
||||
pipeline.connect("prompt_builder_1.prompt", "string_joiner.strings")
|
||||
pipeline.connect("prompt_builder_2.prompt", "string_joiner.strings")
|
||||
|
||||
result = pipeline.run(
|
||||
data={
|
||||
"prompt_builder_1": {"query": string_1},
|
||||
"prompt_builder_2": {"query": string_2},
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
Reference in New Issue
Block a user