chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:22:28 +08:00
commit c56bef871b
9296 changed files with 1854228 additions and 0 deletions
@@ -0,0 +1,8 @@
---
features:
- |
Add run_async for `OpenAITextEmbedder`.
fixes:
- |
`OpenAITextEmbedder` no longer replaces newlines with spaces in the text to embed. This was only required for the
discontinued v1 embedding models.
@@ -0,0 +1,7 @@
---
fixes:
- |
Forward declaration of `AnalyzeResult` type in `AzureOCRDocumentConverter`.
`AnalyzeResult` is already imported in a lazy import block.
The forward declaration avoids issues when `azure-ai-formrecognizer>=3.2.0b2` is not installed.
@@ -0,0 +1,31 @@
---
upgrade:
- |
``AsyncPipeline`` has been removed. Its asynchronous capabilities are now part of the single ``Pipeline``
class, which provides both a synchronous ``run`` method and the asynchronous ``run_async``,
``run_async_generator``, and ``stream`` methods (mirroring how components expose ``run`` and ``run_async``).
To migrate, replace ``AsyncPipeline`` with ``Pipeline``:
.. code-block:: python
# Before
from haystack import AsyncPipeline
pipeline = AsyncPipeline()
result = await pipeline.run_async(data)
# After
from haystack import Pipeline
pipeline = Pipeline()
result = await pipeline.run_async(data)
Serialized pipelines, ``SuperComponent``s, and ``PipelineTool``s are loaded as ``Pipeline`` instances.
``Pipeline.run`` runs synchronously and blocks until completion; in an async context use
``await pipeline.run_async(...)``. The former ``AsyncPipeline.run`` was a synchronous wrapper around the
concurrent engine, so to preserve that behavior use ``asyncio.run(pipeline.run_async(...))``. Note that
``Pipeline.run`` executes components sequentially and does not accept ``concurrency_limit``.
Both synchronous and asynchronous runs are traced under a single ``haystack.pipeline.run`` operation
name, distinguished by a ``haystack.pipeline.execution_mode`` tag (``sync`` or ``async``). Previously,
asynchronous runs used the ``haystack.async_pipeline.run`` operation name.
@@ -0,0 +1,6 @@
---
fixes:
- |
Fixed a bug in ``NamedEntityExtractor`` where the spaCy/Thinc device state was not correctly
restored after execution, potentially affecting the device configuration of other spaCy
components in the same process.
@@ -0,0 +1,7 @@
---
security:
- |
Made QUOTE_SPANS_RE regex ReDoS-safe. This prevents potential catastrophic backtracking on malicious inputs
fixes:
- |
Fixed a potential ReDoS issue in QUOTE_SPANS_RE regex used inside the SentenceSplitter component.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Adapts how ChatPromptBuilder creates ChatMessages. Messages are deep copied to ensure all meta fields are copied correctly.
@@ -0,0 +1,4 @@
---
preview:
- |
Adapt GPTGenerator to use strings for input and output
@@ -0,0 +1,4 @@
---
preview:
- |
Add CohereGenerator compatible with Cohere generate endpoint
@@ -0,0 +1,4 @@
---
preview:
- |
Add MarkdownToTextDocument, a file converter that converts Markdown files into a text Documents.
@@ -0,0 +1,28 @@
---
features:
- |
Add HuggingFace TEI Embedders - `HuggingFaceTEITextEmbedder` and `HuggingFaceTEIDocumentEmbedder`.
An example using `HuggingFaceTEITextEmbedder` to embed a string:
```python
from haystack.components.embedders import HuggingFaceTEITextEmbedder
text_to_embed = "I love pizza!"
text_embedder = HuggingFaceTEITextEmbedder(
model="BAAI/bge-small-en-v1.5", url="<your-tei-endpoint-url>", token="<your-token>"
)
print(text_embedder.run(text_to_embed))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
An example using `HuggingFaceTEIDocumentEmbedder` to create Document embeddings:
```python
from haystack.dataclasses import Document
from haystack.components.embedders import HuggingFaceTEIDocumentEmbedder
doc = Document(content="I love pizza!")
document_embedder = HuggingFaceTEIDocumentEmbedder(
model="BAAI/bge-small-en-v1.5", url="<your-tei-endpoint-url>", token="<your-token>"
)
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
@@ -0,0 +1,8 @@
---
features:
- |
Added an ``after_tool`` hook point to the ``Agent``. Hooks registered under ``after_tool`` run after tools execute,
once their result messages are in ``state.data["messages"]``, and before the exit check and the next LLM call. Use it to
rewrite the freshly produced tool-result messages, for example to offload, redact, truncate, or summarize results
before the next LLM call sees them. It does not run on the plain-text exit step, where no tools run. Register it
like any other hook: ``hooks={"after_tool": [my_hook]}``.
@@ -0,0 +1,129 @@
---
upgrade:
- |
``continue_run`` is now a reserved key in ``Agent.state_schema`` (set by an ``on_exit`` hook to keep the Agent
running). Passing it via the ``state_schema`` argument now raises ``ValueError``. Rename any conflicting state
key (e.g. ``my_continue_run``) to migrate.
features:
- |
Added hooks to the ``Agent``. Pass ``hooks`` as a dictionary mapping a hook point to a list of hooks the Agent runs
at that point. Each hook receives the live ``State`` and influences the run by mutating it in place; hooks for a
hook point run in list order, and the same hook can be registered under multiple hook points. Valid hook points are:
- ``before_llm``: Runs before each chat-generator call.
- ``before_tool``: Runs after the model requests tool calls, before any tools run. After these hooks run, the Agent
re-reads the current last message from ``state["messages"]``. If that message contains tool calls, those calls are
executed. If it does not, no tools run for that step, no tool-based exit condition is triggered, and the Agent
loops back to the next LLM call unless ``max_agent_steps`` has been reached.
- ``on_exit``: Runs when the Agent is about to stop on an exit condition. An ``on_exit`` hook can keep the Agent
running by setting the ``continue_run`` control flag (``state.set("continue_run", True)``), usually alongside a
message telling the model what to do next. ``on_exit`` hooks run when the Agent stops on an exit condition, but
not when it stops because ``max_agent_steps`` is reached.
A hook is any object with a ``run(state)`` method (optionally ``run_async(state)``); use the ``@hook`` decorator
to build one from a function. This enables patterns such as building run-time system context, retrieving memories,
and requiring a tool to be called before finishing.
The example below registers one hook at each placement to show what each can do:
.. code-block:: python
from datetime import datetime, timezone
from typing import Annotated
from haystack.components.agents import Agent
from haystack.components.agents.state import State, replace_values
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.hooks import hook
from haystack.tools import tool
@tool
def search(query: Annotated[str, "The search query"]) -> str:
"""Search the web."""
# Placeholder: would call a real search API
return "Fusion startups reported net-energy-gain milestones this year."
@hook
def build_context(state: State) -> None:
# before_llm: build run-time system context once, before the first model call.
if state.get("step_count") == 0:
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
system = ChatMessage.from_system(f"You are a research assistant. The current time is {now}.")
state.set("messages", [system, *state.data["messages"]], handler_override=replace_values)
@hook
def audit_tool_calls(state: State) -> None:
# before_tool: see which tools the model is about to run.
pending = state.data["messages"][-1].tool_calls
print(f"about to run: {[tc.tool_name for tc in pending]}")
@hook
def require_search(state: State) -> None:
# on_exit: keep going until the agent has actually searched.
if state.get("tool_call_counts", {}).get("search", 0) == 0:
state.set("messages", [ChatMessage.from_system("Search before answering.")])
state.set("continue_run", True)
agent = Agent(
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
tools=[search],
hooks={
"before_llm": [build_context],
"before_tool": [audit_tool_calls],
"on_exit": [require_search],
},
)
result = agent.run(messages=[ChatMessage.from_user("What are the latest developments in fusion energy?")])
print(result["last_message"].text)
Class-based hooks may also implement the optional lifecycle methods ``warm_up`` / ``warm_up_async`` and
``close`` / ``close_async``. The Agent calls them from its own ``warm_up`` / ``warm_up_async`` and
``close`` / ``close_async``, so a hook can defer opening clients or reading credentials until warm-up and release
them on close. When a class-based hook should be serializable, store serializable constructor arguments on the hook
and rebuild runtime clients from those values. For example, an ``on_exit`` hook can grade the Agent's answer with
its own LLM and ask it to improve a weak answer before finishing, warming the judge's client during the Agent's
warm-up:
.. code-block:: python
from typing import Any
from haystack.core.serialization import default_from_dict, default_to_dict
class GradeFinalAnswer:
"""Grade the Agent's answer with an LLM and ask it to improve a weak answer before finishing."""
def __init__(self, model: str = "gpt-5.4-nano"):
self.model = model
self._judge = OpenAIChatGenerator(model=self.model)
def warm_up(self) -> None:
# Warm up the judge's own client during the Agent's warm-up.
self._judge.warm_up()
def run(self, state: State) -> None:
answer = state.data["messages"][-1].text or ""
verdict = self._judge.run(
messages=[ChatMessage.from_user(f"Reply with only PASS or FAIL. Is this answer complete?\n\n{answer}")]
)["replies"][0].text or ""
if "FAIL" in verdict.upper():
state.set("messages", [ChatMessage.from_user("Your answer was incomplete. Please improve it.")])
state.set("continue_run", True)
def to_dict(self) -> dict[str, Any]:
return default_to_dict(self, model=self.model)
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "GradeFinalAnswer":
return default_from_dict(cls, data)
agent = Agent(chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"), hooks={"on_exit": [GradeFinalAnswer()]})
result = agent.run(messages=[ChatMessage.from_user("Explain how vaccines work.")])
print(result["last_message"].text)
@@ -0,0 +1,4 @@
---
preview:
- |
Add the `AnswerBuilder` component for Haystack 2.0 that creates Answer objects from the string output of Generators.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Added support for Apple Silicon GPU acceleration through "mps pytorch", enabling better performance on Apple M1 hardware.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Added the apply_filter_policy function to standardize the application of filter policies across all document store-specific retrievers, allowing for consistent handling of initial and runtime filters based on the chosen policy (replace or merge).
@@ -0,0 +1,5 @@
---
features:
- |
Add `run_async` method to HuggingFaceAPIDocumentEmbedder. This method enriches Documents with embeddings.
It supports the same parameters as the `run` method. It returns a coroutine that can be awaited.
@@ -0,0 +1,20 @@
---
features:
- |
Add native async tool invocation. ``Tool`` now accepts an optional ``async_function`` (a coroutine function) alongside the existing ``function`` field.
When ``Agent.run_async`` invokes a tool, it now awaits the ``async_function`` of Tool if available.
``@tool`` / ``create_tool_from_function`` automatically routes ``async def`` callables to ``async_function``, so simply decorating an ``async def`` produces an async tool:
.. code-block:: python
from typing import Annotated
from haystack.tools import tool
@tool
async def weather(city: Annotated[str, "The name of the city"]) -> str:
"""Get the weather for a city."""
...
``ComponentTool`` automatically wires an async invoker for components that define ``run_async`` (``__haystack_supports_async__`` is ``True``).
``PipelineTool`` enables the async path when wrapping an ``AsyncPipeline`` and falls back to the threaded sync path when wrapping a regular ``Pipeline``.
@@ -0,0 +1,6 @@
---
enhancements:
- |
Added component_name and component_type attributes to PipelineRuntimeError.
- Moved error message creation to within PipelineRuntimeError
- Created a new subclass of PipelineRuntimeError called PipelineComponentsBlockedError for the specific case where the pipeline cannot run since no components are unblocked.
@@ -0,0 +1,41 @@
---
features:
- |
Pipelines now natively support connecting multiple outputs directly to a single component input without requiring
an explicit Joiner component. This only works when the connected outputs and inputs are of compatible list types,
such as ``list[Document]``.
This simplifies pipeline definitions when multiple components produce compatible outputs.
For example, multiple outputs from a ``FileTypeRouter`` can now be connected directly to a single converter or writer, without defining an intermediate ``ListJoiner`` or ``DocumentJoiner``.
.. code:: python
from haystack import Pipeline
from haystack.components.converters import HTMLToDocument, TextFileToDocument
from haystack.components.routers import FileTypeRouter
from haystack.components.writers import DocumentWriter
from haystack.dataclasses import ByteStream
from haystack.document_stores.in_memory import InMemoryDocumentStore
sources = [
ByteStream.from_string(text="Text file content", mime_type="text/plain", meta={"file_type": "txt"}),
ByteStream.from_string(
text="\n<html><body>Some content</body></html>\n", mime_type="text/html", meta={"file_type": "html"},
),
]
doc_store = InMemoryDocumentStore()
pipe = Pipeline()
pipe.add_component("router", FileTypeRouter(mime_types=["text/plain", "text/html"]))
pipe.add_component("txt_converter", TextFileToDocument())
pipe.add_component("html_converter", HTMLToDocument())
pipe.add_component("writer", DocumentWriter(doc_store))
pipe.connect("router.text/plain", "txt_converter.sources")
pipe.connect("router.text/html", "html_converter.sources")
# The DocumentWriter accepts documents from both converters without needing a DocumentJoiner
pipe.connect("txt_converter.documents", "writer.documents")
pipe.connect("html_converter.documents", "writer.documents")
result = pipe.run({"router": {"sources": sources}})
# result["writer"]["documents_written"] == 2
@@ -0,0 +1,5 @@
---
enhancements:
- |
Improved AzureDocumentEmbedder to handle embedding generation failures gracefully.
Errors are logged, and processing continues with the remaining batches.
@@ -0,0 +1,5 @@
---
features:
- |
Adds AzureOpenAIDocumentEmbedder and AzureOpenAITextEmbedder as new embedders. These embedders are very similar to
their OpenAI counterparts, but they use the Azure API instead of the OpenAI API.
@@ -0,0 +1,4 @@
---
features:
- |
Adds support for Azure OpenAI models with AzureOpenAIGenerator and AzureOpenAIChatGenerator components.
@@ -0,0 +1,6 @@
---
enhancements:
- |
Expose default_headers to pass custom headers to Azure API including APIM subscription key.
- |
Add optional azure_kwargs dictionary parameter to pass in parameters undefined in Haystack but supported by AzureOpenAI.
@@ -0,0 +1,24 @@
---
features:
- |
Added the `AzureOpenAIResponsesChatGenerator`, a new component that integrates Azure OpenAI's Responses API into Haystack.
This unlocks several advanced capabilities from the Responses API:
- Allowing retrieval of concise summaries of the model's reasoning process.
- Allowing the use of native OpenAI or MCP tool formats, along with Haystack Tool objects and Toolset instances.
Example with reasoning and web search tool:
```python
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
from haystack.dataclasses import ChatMessage
chat_generator = AzureOpenAIResponsesChatGenerator(
azure_endpoint="https://example-resource.azure.openai.com/",
azure_deployment="gpt-5-mini",
generation_kwargs={"reasoning": {"effort": "low", "summary": "auto"}},
)
response = chat_generator.run(
messages=[ChatMessage.from_user("What's Natural Language Processing?")]
)
print(response["replies"][0].text)
```
@@ -0,0 +1,4 @@
---
preview:
- |
Add AzureOCRDocumentConverter to convert files of different types using Azure's Document Intelligence Service.
@@ -0,0 +1,4 @@
---
features:
- |
Treat bare types (e.g., List, Dict) as generic types with Any arguments during type compatibility checks.
@@ -0,0 +1,5 @@
---
enhancements:
- |
Add batch_size to the __init__ method of FAISS Document Store. This works as the default value for all methods of
FAISS Document Store that support batch_size.
@@ -0,0 +1,5 @@
---
preview:
- |
Add ByteStream type to send binary raw data across components
in a pipeline.
@@ -0,0 +1,14 @@
---
highlights: >
The `Multiplexer` component proved to be hard to explain and to understand. After reviewing its use cases, the documentation
was rewritten and the component was renamed to `BranchJoiner` to better explain its functionalities.
upgrade:
- |
`BranchJoiner` has the very same interface as `Multiplexer`. To upgrade your code, just rename any occurrence
of `Multiplexer` to `BranchJoiner` and ajdust the imports accordingly.
features:
- |
Add `BranchJoiner` to eventually replace `Multiplexer`
deprecations:
- |
`Mulitplexer` is now deprecated.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Add `guess_mime_type` parameter to `Bytestream.from_file_path()`
@@ -0,0 +1,4 @@
---
features:
- Add support for ByteStream objects in MetadataRouter.
It can now be used to route `list[Documents]` or `list[ByteStream]` based on metadata.
@@ -0,0 +1,6 @@
---
features:
- |
Adds `calculate_metrics()` function to EvaluationResult for computation of evaluation metrics.
Adds `Metric` class to store list of available metrics.
Adds `MetricsResult` class to store the metric values computed during the evaluation.
@@ -0,0 +1,4 @@
---
features:
- |
Add compatibility for Callable types.
@@ -0,0 +1,5 @@
---
preview:
- |
Introduce ChatMessage data class to facilitate structured handling and processing of message content
within LLM chat interactions.
@@ -0,0 +1,6 @@
---
fixes:
- |
In the `ChatMessage.to_openai_dict_format` utility method,
include the `name` field in the returned dictionary, if present.
Previously, the `name` field was erroneously skipped.
@@ -0,0 +1,7 @@
---
enhancements:
- |
`ChatPromptBuilder` now supports changing its template at runtime. This allows you to define a default template and then change it based on your needs at runtime.
deprecations:
- |
`DynamicChatPromptBuilder` has been deprecated as `ChatPromptBuilder` fully covers its functionality. Use `ChatPromptBuilder` instead.
@@ -0,0 +1,7 @@
---
features:
- |
- Add a `ComponentInfo` dataclass to the `haystack.dataclasses` module.
This dataclass is used to store information about the component. We pass it to `StreamingChunk` so we can tell from which component a stream is coming from.
- Pass the `component_info` to the `StreamingChunk` in the `OpenAIChatGenerator`, `AzureOpenAIChatGenerator`, `HuggingFaceAPIChatGenerator` and `HuggingFaceLocalChatGenerator`.
@@ -0,0 +1,38 @@
---
highlights: |
Introduced ComponentTool, a powerful addition to the Haystack tooling architecture that enables any Haystack component to be used as a tool by LLMs.
ComponentTool bridges the gap between Haystack's component ecosystem and LLM tool/function calling capabilities, allowing LLMs to
directly interact with components like web search, document processing, or any custom user component. ComponentTool handles
all the complexity of schema generation and type conversion, making it easy to expose component functionality to LLMs.
features:
- |
Introduced the ComponentTool, a new tool that wraps Haystack components allowing them to be utilized as tools for LLMs (various ChatGenerators).
This ComponentTool supports automatic tool schema generation, input type conversion, and offering support for components with run methods that have input types:
- Basic types (str, int, float, bool, dict)
- Dataclasses (both simple and nested structures)
- Lists of basic types (e.g., List[str])
- Lists of dataclasses (e.g., List[Document])
- Parameters with mixed types (e.g., List[Document], str etc.)
Example usage:
```python
from haystack.components.websearch import SerperDevWebSearch
from haystack.tools import ComponentTool
from haystack.utils import Secret
# Create a SerperDev search component
search = SerperDevWebSearch(
api_key=Secret.from_token("your-api-key"),
top_k=3
)
# Create a tool from the component
tool = ComponentTool(
component=search,
name="web_search", # Optional: defaults to "serper_dev_web_search"
description="Search the web for current information" # Optional: defaults to component docstring
)
# You can now use the tool now in a pipeline, see docs for more examples
```
@@ -0,0 +1,4 @@
---
features:
- |
Add a CSV to Document converter component. Loads the file as bytes object. Adds the loaded string as a new document that can be used for further processing by the Document Splitter.
@@ -0,0 +1,16 @@
---
enhancements:
- |
Allow the ability to add the current date inside a template in `PromptBuilder` using the following syntax:
- `{% now 'UTC' %}`: Get the current date for the UTC timezone.
- `{% now 'America/Chicago' + 'hours=2' %}`: Add two hours to the current date in the Chicago timezone.
- `{% now 'Europe/Berlin' - 'weeks=2' %}`: Subtract two weeks from the current date in the Berlin timezone.
- `{% now 'Pacific/Fiji' + 'hours=2', '%H' %}`: Display only the number of hours after adding two hours to the Fiji timezone.
- `{% now 'Etc/GMT-4', '%I:%M %p' %}`: Change the date format to AM/PM for the GMT-4 timezone.
Note that if no date format is provided, the default will be `%Y-%m-%d %H:%M:%S`. Please refer to [list of tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) for a list of timezones.
@@ -0,0 +1,4 @@
---
preview:
- |
Add callable hook to PyPDFToDocument to enable easier customization of pdf to Document conversion.
@@ -0,0 +1,6 @@
---
features:
- |
Added custom filters support to ConditionalRouter. Users can now pass in
one or more custom Jinja2 filter callables and be able to access those
filters when defining condition expressions in routes.
@@ -0,0 +1,12 @@
---
features:
- |
We've added a new **DALLEImageGenerator** component, bringing image generation with OpenAI's DALL-E to the Haystack
- **Easy to Use**: Just a few lines of code to get started:
```python
from haystack.components.generators import DALLEImageGenerator
image_generator = DALLEImageGenerator()
response = image_generator.run("Show me a picture of a black cat.")
print(response)
```
@@ -0,0 +1,6 @@
---
fixes:
- |
Add dataframe to legacy fields for the Document dataclass.
This fixes a bug where Document.from_dict() in haystack-ai>=2.11.0 could not properly deserialize a Document
dictionary obtained with document.to_dict(flatten=False) in haystack-ai<=2.10.0.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Added `default_headers` parameter to `AzureOpenAIDocumentEmbedder` and `AzureOpenAITextEmbedder`.
@@ -0,0 +1,5 @@
---
enhancements:
- |
Add a field called `default_value` to the `InputSocket` dataclass.
Derive `is_mandatory` value from the presence of `default_value`.
@@ -0,0 +1,4 @@
---
deprecations:
- |
The output of the ContextRelevanceEvaluator will change in Haystack 2.4.0. Contexts will be scored as a whole instead of individual statements and only the relevant sentences will be returned. A score of 1 is now returned if a relevant sentence is found, and 0 otherwise.
@@ -0,0 +1,4 @@
---
enhancements:
- |
add dimensions parameter to Azure OpenAI Embedders (AzureOpenAITextEmbedder and AzureOpenAIDocumentEmbedder) to fully support new embedding models like text-embedding-3-small, text-embedding-3-large and upcoming ones
@@ -0,0 +1,4 @@
---
enhancements:
- |
add dimensions parameter to OpenAI Embedders to fully support new embedding models like text-embedding-3-small, text-embedding-3-large and upcoming ones
@@ -0,0 +1,6 @@
---
features:
- |
Added a new mode in JoinDocuments,
Distribution-based rank fusion as
[the article](https://medium.com/plain-simple-software/distribution-based-score-fusion-dbsf-a-new-approach-to-vector-search-ranking-f87c37488b18)
@@ -0,0 +1,6 @@
---
features:
- |
Add `SentenceTransformersDiversityRanker`.
The Diversity Ranker orders documents in such a way as to maximize the overall diversity of the given documents.
The ranker leverages sentence-transformer models to calculate semantic embeddings for each document and the query.
@@ -0,0 +1,16 @@
upgrade:
- |
``DocumentNDCGEvaluator`` now matches documents by their ``content`` field
by default instead of their auto-generated ``id``. Previously, ground
truth and retrieved documents were matched only if they had identical
``id`` values, which rarely happened in practice since IDs are generated
independently for each Document instance. As a result, NDCG scores
computed with this evaluator may change for existing pipelines. To keep
the previous ``id``-based matching behavior, pass
``document_comparison_field="id"`` when constructing the evaluator.
enhancements:
- |
Added ``document_comparison_field`` parameter to ``DocumentNDCGEvaluator``,
consistent with ``DocumentMAPEvaluator``, ``DocumentMRREvaluator``, and
``DocumentRecallEvaluator``. Users can now match documents by ``"content"``,
``"id"``, or any ``"meta.<key>"`` field when calculating NDCG scores.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Document writer returns the number of documents written.
@@ -0,0 +1,6 @@
---
highlights: >
Adding the `DocxToDocument` component to convert Docx files to Documents.
features:
- |
Adding the `DocxToDocument` component inside the `converters` category. It uses the `python-docx` library to convert Docx files to haystack Documents.
@@ -0,0 +1,4 @@
---
preview:
- |
Adds `ChatMessage` templating in `PromptBuilder`
@@ -0,0 +1,5 @@
---
preview:
- |
Add `DynamicPromptBuilder` to dynamically generate prompts from either a list of ChatMessage instances or a string
template, leveraging Jinja2 templating for flexible and efficient prompt construction.
@@ -0,0 +1,7 @@
---
features:
- |
Added the `enable_streaming_callback_passthrough` to the `ToolInovker` init, run and run_async methods. If set to True the ToolInvoker will try and pass the `streaming_callback` function to a tool's invoke method only if the tool's invoke method has `streaming_callback` in its signature.
fixes:
- |
Fixed the `to_dict` and `from_dict` of `ToolInvoker` to properly serialize the `streaming_callback` init parameter.
@@ -0,0 +1,6 @@
---
enhancements:
- |
Enhanced `SentenceTransformersDocumentEmbedder` and `SentenceTransformersTextEmbedder` to accept
an additional parameter, which is passed directly to the underlying `SentenceTransformer.encode` method
for greater flexibility in embedding customization.
@@ -0,0 +1,4 @@
---
fixes:
- |
Add encoding format keyword argument to OpenAI client when creating embeddings.
@@ -0,0 +1,4 @@
---
features:
- |
Updated our serialization and deserialization of PipelineSnapshots to work with python Enum classes.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Improved error handling for component `run` failures by raising a runtime error that includes the component's name and type.
@@ -0,0 +1,4 @@
preview:
- |
Add eval function for evaluation of components and Pipelines.
Adds EvaluationResult to store results of evaluation.
@@ -0,0 +1,8 @@
---
features:
- |
Adds support for the Exact Match metric to `EvaluationResult.calculate_metrics(...)`:
```python
from haystack.evaluation.metrics import Metric
exact_match_metric = eval_result.calculate_metrics(Metric.EM, output_key="answers")
```
@@ -0,0 +1,4 @@
---
features:
- |
Add XLSXToDocument converter that loads an Excel file using Pandas + openpyxl and by default converts each sheet into a separate Document in a CSV format.
@@ -0,0 +1,4 @@
---
features:
- |
Add an `extra` field to `ToolCall` and `ToolCallDelta` to store provider-specific information.
@@ -0,0 +1,8 @@
---
features:
- |
Adds support for the F1 metric to `EvaluationResult.calculate_metrics(...)`:
```python
from haystack.evaluation.metrics import Metric
f1_metric = eval_result.calculate_metrics(Metric.F1, output_key="answers")
```
@@ -0,0 +1,5 @@
---
enhancements:
- |
If an LLM-based evaluator (e.g., `Faithfulness` or `ContextRelevance`) is initialised with `raise_on_failure=False`, and if a call to an LLM fails or an LLM outputs an invalid JSON, the score of the sample is set to `NaN` instead of raising an exception.
The user is notified with a warning indicating the number of requests that failed.
@@ -0,0 +1,6 @@
---
highlights: >
Introduced `FallbackChatGenerator` that tries multiple chat providers one by one, improving reliability in production and making sure you get answers even when some provider fails.
features:
- |
Added `FallbackChatGenerator` that automatically retries different chat generators and returns first successful response with detailed information about which providers were tried.
@@ -0,0 +1,4 @@
---
preview:
- |
Adds FileExtensionClassifier to preview components.
@@ -0,0 +1,5 @@
---
features:
- |
Add FilterRetriever.
It retrieves documents that match the provided (either at init or runtime) filters.
@@ -0,0 +1,6 @@
---
features:
- |
Added dedicated `finish_reason` field to `StreamingChunk` class to improve type safety and enable sophisticated streaming UI logic. The field uses a `FinishReason` type alias with standard values: "stop", "length", "tool_calls", "content_filter", plus Haystack-specific value "tool_call_results" (used by ToolInvoker to indicate tool execution completion).
- |
Updated `ToolInvoker` component to use the new `finish_reason` field when streaming tool results. The component now sets `finish_reason="tool_call_results"` in the final streaming chunk to indicate that tool execution has completed, while maintaining backward compatibility by also setting the value in `meta["finish_reason"]`.
@@ -0,0 +1,8 @@
---
enhancements:
- |
Added ``haystack.component.fully_qualified_type`` field to component tracing output.
This new field provides the full module path and class name (e.g.,
``haystack.components.generators.chat.openai.OpenAIChatGenerator``) alongside the
existing ``haystack.component.type`` field that only contains the class name.
This enables dynamic component loading and better tooling integration.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Adds `generation_kwargs` to the `Agent` component, allowing for more fine-grained control at run-time over the chat generation.
@@ -0,0 +1,7 @@
---
enhancements:
- |
Add async variants of metadata methods to ``InMemoryDocumentStore``:
``get_metadata_fields_info_async()``, ``get_metadata_field_min_max_async()``, and
``get_metadata_field_unique_values_async()``. These rely on the store's thread-pool executor,
consistent with the existing async method pattern.
@@ -0,0 +1,4 @@
---
features:
- |
Added haystack-experimental to the project's dependencies to enable automatic use of cutting-edge features from Haystack. Users can now access components from haystack-experimental by simply importing them from haystack_experimental instead of haystack. For more information, visit https://github.com/deepset-ai/haystack-experimental.
@@ -0,0 +1,7 @@
---
fixes:
- |
Resolves a bug where the HuggingFaceTGIGenerator and HuggingFaceTGIChatGenerator encountered issues if provided
with valid models that were not available on the HuggingFace inference API rate-limited tier. The fix, detailed
in [GitHub issue #6816](https://github.com/deepset-ai/haystack/issues/6816) and its GitHub PR, ensures these
components now correctly handle model availability, eliminating previous limitations.
@@ -0,0 +1,32 @@
---
features:
- |
Added support for human confirmation on ``Agent`` tool calls.
You can now configure per-tool confirmation strategies when building an ``Agent``, including always requiring confirmation,
never requiring it, or requesting confirmation only on first use.
The confirmation experience is customizable through different UI strategies, allowing fine-grained control over how and when users approve tool executions.
.. code-block:: python
agent = Agent(
chat_generator=OpenAIChatGenerator(model="gpt-4.1"),
tools=[balance_tool, addition_tool, phone_tool],
system_prompt=(
"You are a helpful financial assistant. "
"Use the provided tool to get bank balances when needed."
),
confirmation_strategies={
balance_tool.name: BlockingConfirmationStrategy(
confirmation_policy=AlwaysAskPolicy(),
confirmation_ui=RichConsoleUI(console=cons),
),
addition_tool.name: BlockingConfirmationStrategy(
confirmation_policy=NeverAskPolicy(),
confirmation_ui=SimpleConsoleUI(),
),
phone_tool.name: BlockingConfirmationStrategy(
confirmation_policy=AskOncePolicy(),
confirmation_ui=SimpleConsoleUI(),
),
},
)
@@ -0,0 +1,4 @@
---
preview:
- |
Adds HTMLToDocument component to convert HTML to a Document.
@@ -0,0 +1,19 @@
---
features:
- |
Introducing the HuggingFaceLocalChatGenerator, a new chat-based generator designed for leveraging chat models from
Hugging Face's (HF) model hub. Users can now perform inference with chat-based models in a local runtime, utilizing
familiar HF generation parameters, stop words, and even employing custom chat templates for custom message formatting.
This component also supports streaming responses and is optimized for compatibility with a variety of devices.
Here is an example of how to use the HuggingFaceLocalChatGenerator:
```python
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
from haystack.dataclasses import ChatMessage
generator = HuggingFaceLocalChatGenerator(model="HuggingFaceH4/zephyr-7b-beta")
generator.warm_up()
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
print(generator.run(messages))
```
@@ -0,0 +1,5 @@
---
features:
- |
Added new `HuggingFaceTEIRanker` component to enable reranking with Text Embeddings Inference (TEI) API.
This component supports both self-hosted Text Embeddings Inference services and Hugging Face Inference Endpoints.
@@ -0,0 +1,5 @@
---
preview:
- |
Adds `HuggingFaceTGIChatGenerator` for text and chat generation. This components support remote inferencing for
Hugging Face LLMs via text-generation-inference (TGI) protocol.
@@ -0,0 +1,5 @@
---
preview:
- |
Adds `HuggingFaceTGIGenerator` for text generation. This components support remote inferencing for
Hugging Face LLMs via text-generation-inference (TGI) protocol.
@@ -0,0 +1,13 @@
---
features:
- |
Added ``link_format`` parameter to ``PPTXToDocument`` and ``XLSXToDocument`` converters,
allowing extraction of hyperlink addresses from PPTX and XLSX files.
Supported formats:
- ``"markdown"``: ``[text](url)``
- ``"plain"``: ``text (url)``
- ``"none"`` (default): Only text is extracted, link addresses are ignored.
This follows the same pattern already available in ``DOCXToDocument``.
@@ -0,0 +1,6 @@
---
enhancements:
- |
Added multimodal support to `HuggingFaceAPIChatGenerator` to enable vision-language model (VLM) usage with images and text.
Users can now send both text and images to VLM models through Hugging Face APIs. The implementation follows the HF VLM API format
specification and maintains full backward compatibility with text-only messages.
@@ -0,0 +1,4 @@
---
features:
- |
Add a indexing `build_indexing_pipeline` utility function
@@ -0,0 +1,4 @@
---
enhancements:
- |
Adds inference mode to model call of the ExtractiveReader. This prevents gradients from being calculated during inference time in pytorch.
@@ -0,0 +1,5 @@
---
enhancements:
- |
Added a new parameter to `EvaluationRunResult.comparative_individual_scores_report()` to specify columns to keep in the comparative DataFrame.
@@ -0,0 +1,4 @@
---
enhancements:
- |
The `DocumentCleaner` class has the optional attribute `keep_id` that if set to True it keeps the document ids unchanged after cleanup.
@@ -0,0 +1,5 @@
---
preview:
- |
Introduced the LinkContentFetcher in Haystack 2.0. This component fetches content from specified
URLs and converts them into ByteStream objects for further processing in Haystack pipelines.
@@ -0,0 +1,4 @@
---
features:
- |
Added a new component `ListJoiner` which joins lists of values from different components to a single list.
@@ -0,0 +1,26 @@
---
features:
- |
Added a new ``LLM`` component (``haystack.components.generators.chat.LLM``) that provides a simplified
interface for text generation powered by a large language model. The ``LLM`` component is a streamlined
version of the ``Agent`` that focuses solely on single-turn text generation without tool usage.
It supports system prompts, templated user prompts with required variables, streaming callbacks,
and both synchronous (``run``) and asynchronous (``run_async``) execution.
Usage example:
.. code:: python
from haystack.components.generators.chat import LLM
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
llm = LLM(
chat_generator=OpenAIChatGenerator(),
system_prompt="You are a helpful translation assistant.",
user_prompt=\"\"\"{% message role="user"%}
Summarize the following document: {{ document }}
{% endmessage %}\"\"\",
required_variables=["document"],
)
result = llm.run(document="The weather is lovely today and the sun is shining. ")
print(result["last_message"].text)
@@ -0,0 +1,5 @@
---
features:
- |
Added the LLMDocumentContentExtractor which extracts textual content from image-based documents using a vision-enabled LLM.
This is good for creating a textual representation of an image which can be used when only text-based retrieval is available.
@@ -0,0 +1,5 @@
features:
- |
Added ``LLMRanker``, a new ranker component that uses a ``ChatGenerator`` and ``PromptBuilder`` to rerank
documents based on JSON-formatted LLM output. ``LLMRanker`` supports configurable prompts, optional custom
chat generators, runtime ``top_k`` overrides, and serialization.
@@ -0,0 +1,4 @@
---
features:
- |
If logprobs are enabled in the generation kwargs, return logprobs in ChatMessage.meta for `OpenAIChatGenerator` and `OpenAIResponsesChatGenerator`.
@@ -0,0 +1,7 @@
---
features:
- |
Add warning logs to the PDFMinerToDocument and PyPDFToDocument to indicate when a processed PDF file has no content.
This can happen if the PDF file is a scanned image.
Also added an explicit check and warning message to the DocumentSplitter that warns the user that empty Documents are skipped.
This behavior was already occurring, but now its clearer through logs that this is happening.
@@ -0,0 +1,18 @@
---
prelude: >
We're excited to introduce a new ranker to Haystack - LostInTheMiddleRanker.
It reorders documents based on the "Lost in the Middle" order, a strategy that
places the most relevant paragraphs at the beginning or end of the context,
while less relevant paragraphs are positioned in the middle. This ranker,
based on the research paper "Lost in the Middle: How Language Models Use Long
Contexts" by Liu et al., can be leveraged in Retrieval-Augmented Generation
(RAG) pipelines.
features:
- |
The LostInTheMiddleRanker can be used like other rankers in Haystack. After
initializing LostInTheMiddleRanker with the desired parameters, it can be
used to rank/reorder a list of documents based on the "Lost in the Middle"
order - the most relevant documents are located at the top and bottom of
the returned list, while the least relevant documents are found in the
middle. We advise that you use this ranker in combination with other rankers,
and to place it towards the end of the pipeline.
@@ -0,0 +1,8 @@
---
features:
- |
Add LostInTheMiddleRanker.
It reorders documents based on the "Lost in the Middle" order, a strategy that
places the most relevant paragraphs at the beginning or end of the context,
while less relevant paragraphs are positioned in the middle.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Adds markdown mimetype support to the file type router i.e. `FileTypeRouter` class.
@@ -0,0 +1,4 @@
---
enhancements:
- |
Added the Maximum Margin Relevance (MMR) strategy to the `SentenceTransformersDiversityRanker`. MMR scores are calculated for each document based on their relevance to the query and diversity from already selected documents.
@@ -0,0 +1,8 @@
---
features:
- |
Introduced the ``MarkdownHeaderSplitter`` component:
- Splits documents into chunks at Markdown headers (``#``, ``##``, etc.), preserving header hierarchy as metadata.
- Supports secondary splitting (by word, passage, period, or line) for further chunking after header-based splitting using Haystack's ``DocumentSplitter``.
- Preserves and propagates metadata such as parent headers and page numbers.
- Handles edge cases such as documents with no headers, empty content, and non-text documents.

Some files were not shown because too many files have changed in this diff Show More