chore: import upstream snapshot with attribution

docs-website/versioned_docs/version-2.22/optimization/advanced-rag-techniques.mdx

@@ -0,0 +1,19 @@
+---
+title: "Advanced RAG Techniques"
+id: advanced-rag-techniques
+slug: "/advanced-rag-techniques"
+---
+
+# Advanced RAG Techniques
+
+This section of documentation talks about advanced RAQ techniques you can implement with Haystack.
+
+Read more about [Hypothetical Document Embeddings (HyDE)](advanced-rag-techniques/hypothetical-document-embeddings-hyde.mdx),
+
+or check out one of our cookbooks 🧑‍🍳:
+
+- [Using Hypothetical Document Embedding (HyDE) to Improve Retrieval](https://haystack.deepset.ai/cookbook/using_hyde_for_improved_retrieval)
+- [Query Decomposition and Reasoning](https://haystack.deepset.ai/cookbook/query_decomposition)
+- [Improving Retrieval by Embedding Meaningful Metadata](https://haystack.deepset.ai/cookbook/improve-retrieval-by-embedding-metadata)
+- [Query Expansion](https://haystack.deepset.ai/cookbook/query-expansion)
+- [Automated Structured Metadata Enrichment](https://haystack.deepset.ai/cookbook/metadata_enrichment)

docs-website/versioned_docs/version-2.22/optimization/advanced-rag-techniques/hypothetical-document-embeddings-hyde.mdx

@@ -0,0 +1,110 @@
+---
+title: "Hypothetical Document Embeddings (HyDE)"
+id: hypothetical-document-embeddings-hyde
+slug: "/hypothetical-document-embeddings-hyde"
+description: "Enhance the retrieval in Haystack using HyDE method by generating a mock-up hypothetical document for an initial query."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Hypothetical Document Embeddings (HyDE)
+
+Enhance the retrieval in Haystack using HyDE method by generating a mock-up hypothetical document for an initial query.
+
+## When Is It Helpful?
+
+The HyDE method is highly useful when:
+
+- The performance of the retrieval step in your pipeline is not good enough (for example, low Recall metric).
+- Your retrieval step has a query as input and returns documents from a larger document base.
+- Particularly worth a try if your data (documents or queries) come from a special domain that is very different from the typical datasets that Retrievers are trained on.
+
+## How Does It Work?
+
+Many embedding retrievers generalize poorly to new, unseen domains. This approach tries to tackle this problem. Given a query, the Hypothetical Document Embeddings (HyDE) first zero-shot prompts an instruction-following language model to generate a “fake” hypothetical document that captures relevant textual patterns from the initial query - in practice, this is done five times. Then, it encodes each hypothetical document into an embedding vector and averages them. The resulting, single embedding can be used to identify a neighbourhood in the document embedding space from which similar actual documents are retrieved based on vector similarity. As with any other retriever, these retrieved documents can then be used downstream in a pipeline (for example, in a Generator for RAG). Refer to the paper “[Precise Zero-Shot Dense Retrieval without Relevance Labels](https://aclanthology.org/2023.acl-long.99/)” for more details.
+<ClickableImage src="/img/2d00628-Untitled_2.png" alt="HyDE model architecture diagram showing how GPT generates hypothetical documents from queries in multiple languages, which are then matched with real documents via a Contriever model" size="large" />
+
+## How To Build It in Haystack?
+
+First, prepare all the components that you would need:
+
+```python
+import os
+from numpy import array, mean
+from typing import List
+
+from haystack.components.generators.openai import OpenAIGenerator
+from haystack.components.builders import PromptBuilder
+from haystack import component, Document
+from haystack.components.converters import OutputAdapter
+from haystack.components.embedders import SentenceTransformersDocumentEmbedder
+
+## We need to ensure we have the OpenAI API key in our environment variables
+os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_KEY"
+
+## Initializing standard Haystack components
+generator = OpenAIGenerator(
+    model="gpt-3.5-turbo",
+    generation_kwargs={"n": 5, "temperature": 0.75, "max_tokens": 400},
+)
+prompt_builder = PromptBuilder(
+    template="""Given a question, generate a paragraph of text that answers the question.    Question: {{question}}    Paragraph:""",
+)
+
+adapter = OutputAdapter(
+    template="{{answers | build_doc}}",
+    output_type=List[Document],
+    custom_filters={"build_doc": lambda data: [Document(content=d) for d in data]},
+)
+
+embedder = SentenceTransformersDocumentEmbedder(
+    model="sentence-transformers/all-MiniLM-L6-v2",
+)
+embedder.warm_up()
+
+
+## Adding one custom component that returns one, "average" embedding from multiple (hypothetical) document embeddings
+@component
+class HypotheticalDocumentEmbedder:
+    @component.output_types(hypothetical_embedding=List[float])
+    def run(self, documents: List[Document]):
+        stacked_embeddings = array([doc.embedding for doc in documents])
+        avg_embeddings = mean(stacked_embeddings, axis=0)
+        hyde_vector = avg_embeddings.reshape((1, len(avg_embeddings)))
+        return {"hypothetical_embedding": hyde_vector[0].tolist()}
+```
+
+Then, assemble them all into a pipeline:
+
+```python
+from haystack import Pipeline
+
+pipeline = Pipeline()
+pipeline.add_component(name="prompt_builder", instance=prompt_builder)
+pipeline.add_component(name="generator", instance=generator)
+pipeline.add_component(name="adapter", instance=adapter)
+pipeline.add_component(name="embedder", instance=embedder)
+pipeline.add_component(name="hyde", instance=HypotheticalDocumentEmbedder())
+
+pipeline.connect("prompt_builder", "generator")
+pipeline.connect("generator.replies", "adapter.answers")
+pipeline.connect("adapter.output", "embedder.documents")
+pipeline.connect("embedder.documents", "hyde.documents")
+query = "What should I do if I have a fever?"
+result = pipeline.run(data={"prompt_builder": {"question": query}})
+
+## 'hypothetical_embedding': [0.0990725576877594, -0.017647066991776227, 0.05918873250484467, ...]}
+```
+
+Here's the graph of the resulting pipeline:
+<ClickableImage src="/img/74f3daa-hyde.png" alt="HyDE pipeline implementation flowchart showing prompt builder, generator, adapter, embedder, and hypothetical document embedder components" size="large"/>
+
+This pipeline example turns your query into one embedding.
+
+You can continue and feed this embedding to any [Embedding Retriever](../../pipeline-components/retrievers.mdx#dense-embedding-based-retrievers) to find similar documents in your Document Store.
+
+## Additional References
+
+📚 Article: [Optimizing Retrieval with HyDE](https://haystack.deepset.ai/blog/optimizing-retrieval-with-hyde)
+
+🧑‍🍳 Cookbook: [Using Hypothetical Document Embedding (HyDE) to Improve Retrieval](https://haystack.deepset.ai/cookbook/using_hyde_for_improved_retrieval)

docs-website/versioned_docs/version-2.22/optimization/evaluation.mdx

@@ -0,0 +1,64 @@
+---
+title: "Evaluation"
+id: evaluation
+slug: "/evaluation"
+description: "Learn all about pipeline or component evaluation in Haystack."
+---
+
+# Evaluation
+
+Learn all about pipeline or component evaluation in Haystack.
+
+Haystack has all the tools needed to evaluate entire pipelines or individual components like Retrievers, Readers, or Generators. This guide explains how to evaluate your pipeline in different scenarios and how to understand the metrics.
+
+Use evaluation and its results to:
+
+- Judge how well your system is performing on a given domain,
+- Compare the performance of different models,
+- Identify underperforming components in your pipeline.
+
+## Evaluation Options
+
+**Evaluating individual components or end-to-end pipelines.**
+
+Evaluating individual components can help understand performance bottlenecks and optimize one component at a time, for example, a Retriever or a prompt used with a Generator.
+
+End-to-end evaluation checks how the full pipeline is used and evaluates only the final outputs. The pipeline is approached as a black box.
+
+**Using ground-truth labels or no labels at all.**
+
+Most statistical evaluators require ground truth labels, such as the documents relevant to the query or the expected answer. In contrast, most model-based evaluators work without any labels just by following the prompt instructions. However, few-shot labels included in the prompt can improve the evaluator.
+
+**Model-based evaluation using a language model or statistical evaluation.**
+
+Model-based evaluation uses LLMs with prompt instructions or smaller fine-tuned models to score aspects of a pipeline’s outputs. Statistical evaluation requires no models and is thus a more lightweight way to score pipeline outputs. For more information, see our docs on [model-based](evaluation/model-based-evaluation.mdx) evaluation and [statistical](evaluation/statistical-evaluation.mdx) evaluation.
+
+## Evaluator Components
+
+|  |  |  |  |
+| --- | --- | --- | --- |
+| Evaluator                                                    | Evaluates Answers or Documents | Model-based or Statistical | Requires Labels |
+| [AnswerExactMatchEvaluator](../pipeline-components/evaluators/answerexactmatchevaluator.mdx) | Answers                        | Statistical                | Yes             |
+| [ContextRelevanceEvaluator](../pipeline-components/evaluators/contextrelevanceevaluator.mdx) | Documents                      | Model-based                | No              |
+| [DocumentMRREvaluator](../pipeline-components/evaluators/documentmrrevaluator.mdx)           | Documents                      | Statistical                | Yes             |
+| [DocumentMAPEvaluator](../pipeline-components/evaluators/documentmapevaluator.mdx)           | Documents                      | Statistical                | Yes             |
+| [DocumentRecallEvaluator](../pipeline-components/evaluators/documentrecallevaluator.mdx)     | Documents                      | Statistical                | Yes             |
+| [FaithfulnessEvaluator](../pipeline-components/evaluators/faithfulnessevaluator.mdx)         | Answers                        | Model-based                | No              |
+| [LLMEvaluator](../pipeline-components/evaluators/llmevaluator.mdx)                           | User-defined                   | Model-based                | No              |
+| [SASEvaluator](../pipeline-components/evaluators/sasevaluator.mdx)                           | Answers                        | Model-based                | Yes             |
+
+## Evaluator Integrations
+
+To learn more about our integration with the Ragas and DeepEval evaluation frameworks, head over to the [RagasEvaluator](../pipeline-components/evaluators/ragasevaluator.mdx) and [DeepEvalEvaluator](../pipeline-components/evaluators/deepevalevaluator.mdx) component docs.
+
+To get started using practical examples, check out our evaluation tutorial or the respective cookbooks below.
+
+## Additional References
+
+:notebook: Tutorial: [Evaluating RAG Pipelines](https://haystack.deepset.ai/tutorials/35_evaluating_rag_pipelines)
+
+🧑‍🍳 Cookbooks:
+
+- [RAG Evaluation with Prometheus 2](https://haystack.deepset.ai/cookbook/prometheus2_evaluation)
+- [RAG Pipeline Evaluation Using Ragas](https://haystack.deepset.ai/cookbook/rag_eval_ragas)
+- [RAG Pipeline Evaluation Using DeepEval](https://haystack.deepset.ai/cookbook/rag_eval_deep_eval)

docs-website/versioned_docs/version-2.22/optimization/evaluation/model-based-evaluation.mdx

@@ -0,0 +1,127 @@
+---
+title: "Model-Based Evaluation"
+id: model-based-evaluation
+slug: "/model-based-evaluation"
+description: "Haystack supports various kinds of model-based evaluation. This page explains what model-based evaluation is and discusses the various options available with Haystack."
+---
+
+# Model-Based Evaluation
+
+Haystack supports various kinds of model-based evaluation. This page explains what model-based evaluation is and discusses the various options available with Haystack.
+
+## What is Model-Based Evaluation
+
+Model-based evaluation in Haystack uses a language model to check the results of a Pipeline. This method is easy to use because it usually doesn't need labels for the outputs. It's often used with Retrieval-Augmented Generative (RAG) Pipelines, but can work with any Pipeline.
+
+Currently, Haystack supports the end-to-end, model-based evaluation of a complete RAG Pipeline.
+
+### Using LLMs for Evaluation
+
+A common strategy for model-based evaluation involves using a Language Model (LLM), such as OpenAI's GPT models, as the evaluator model, often referred to as the _golden_ model. The most frequently used golden model is GPT-4. We utilize this model to evaluate a RAG Pipeline by providing it with the Pipeline's results and sometimes additional information, along with a prompt that outlines the evaluation criteria.
+
+This method of using an LLM as the evaluator is very flexible as it exposes a number of metrics to you. Each of these metrics is ultimately a well-crafted prompt describing to the LLM how to evaluate and score results. Common metrics are faithfulness, context relevance, and so on.
+
+### Using Local LLMs
+
+To use the model-based Evaluators with a local model, you need to pass the `api_base_url` and `model` in the `api_params` parameter when initializing the Evaluator.
+
+The following example shows how this would work with an Ollama model.
+
+First, make sure that Ollama is running locally:
+
+```curl
+curl http://localhost:11434/api/generate -d '{
+  "model": "llama3",
+  "prompt":"Why is the sky blue?"
+}'
+```
+
+Then, your pipeline would look like this:
+
+```python
+from haystack.components.evaluators import FaithfulnessEvaluator
+from haystack.utils import Secret
+
+questions = ["Who created the Python language?"]
+contexts = [
+    [
+        (
+            "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming "
+            "language. Its design philosophy emphasizes code readability, and its language constructs aim to help "
+            "programmers write clear, logical code for both small and large-scale software projects."
+        ),
+    ],
+]
+predicted_answers = [
+    "Python is a high-level general-purpose programming language that was created by George Lucas.",
+]
+local_endpoint = "http://localhost:11434/v1"
+
+evaluator = FaithfulnessEvaluator(
+    api_key=Secret.from_token("just-a-placeholder"),
+    api_params={"api_base_url": local_endpoint, "model": "llama3"},
+)
+
+result = evaluator.run(
+    questions=questions,
+    contexts=contexts,
+    predicted_answers=predicted_answers,
+)
+```
+
+### Using Small Cross-Encoder Models for Evaluation
+
+Alongside LLMs for evaluation, we can also use small cross-encoder models. These models can calculate, for example, semantic answer similarity. In contrast to metrics based on LLMs, the metrics based on smaller models don’t require an API key of a model provider.
+
+This method of using small cross-encoder models as evaluators is faster and cheaper to run but is less flexible in terms of what aspect you can evaluate. You can only evaluate what the small model was trained to evaluate.
+
+## Model-Based Evaluation Pipelines in Haystack
+
+There are two ways of performing model-based evaluation in Haystack, both of which leverage [Pipelines](../../concepts/pipelines.mdx) and [Evaluator](../../pipeline-components/evaluators.mdx) components.
+
+- You can create and run an evaluation Pipeline independently. This means you’ll have to provide the required inputs to the evaluation Pipeline manually. We recommend this way because the separation of your RAG Pipeline and your evaluation Pipeline allows you to store the results of your RAG Pipeline and try out different evaluation metrics afterward without needing to re-run your RAG Pipeline every time.
+- As another option, you can add an evaluator component to the end of a RAG Pipeline. This means you run both a RAG Pipeline and evaluation on top of it in a single `pipeline.run()`  call.
+
+### Model-based Evaluation of Retrieved Documents
+
+#### [ContextRelevanceEvaluator](../../pipeline-components/evaluators/contextrelevanceevaluator.mdx)
+
+Context relevance refers to how relevant the retrieved documents are to the query. An LLM is used to judge that aspect. It first extracts statements from the documents and then checks how many of them are relevant for answering the query.
+
+### Model-based Evaluation of Generated or Extracted Answers
+
+#### [FaithfulnessEvaluator](../../pipeline-components/evaluators/faithfulnessevaluator.mdx)
+
+Faithfulness, also called groundedness, evaluates to what extent a generated answer is based on retrieved documents. An LLM is used to extract statements from the answer and check the faithfulness for each separately. If the answer is not based on the documents, the answer, or at least parts of it, is called a hallucination.
+
+#### [SASEvaluator](../../pipeline-components/evaluators/sasevaluator.mdx) (Semantic Answer Similarity)
+
+Semantic answer similarity uses a transformer-based, cross-encoder architecture to evaluate the semantic similarity of two answers rather than their lexical overlap. While F1 and EM would both score _one hundred percent_ as sharing zero similarity with _100 %_, SAS is trained to assign a high score to such cases. SAS is particularly useful for seeking out cases where F1 doesn't give a good indication of the validity of a predicted answer. You can read more about SAS in [Semantic Answer Similarity for Evaluating Question-Answering Models paper](https://arxiv.org/abs/2108.06130).
+
+### Evaluation Framework Integrations
+
+Currently, Haystack has integrations with [DeepEval](https://docs.confident-ai.com/docs/metrics-introduction) and [Ragas](https://docs.ragas.io/en/stable/index.html). There is an Evaluator component available for each of these frameworks:
+
+- [RagasEvaluator](../../pipeline-components/evaluators/ragasevaluator.mdx)
+- [DeepEvalEvaluator](../../pipeline-components/evaluators/deepevalevaluator.mdx)
+
+|  |  |  |
+| --- | --- | --- |
+| Feature/Integration | RagasEvaluator | DeepEvalEvaluator |
+| Evaluator Models | All GPT models from OpenAI  <br />Google VertexAI Models  <br />Azure OpenAI Models  <br />Amazon Bedrock Models | All GPT models from OpenAI |
+| Supported metrics | ANSWER_CORRECTNESS, FAITHFULNESS, ANSWER_SIMILARITY, CONTEXT_PRECISION, CONTEXT_UTILIZATION,CONTEXT_RECALL, ASPECT_CRITIQUE, CONTEXT_RELEVANCY, ANSWER_RELEVANCY | ANSWER_RELEVANCY, FAITHFULNESS, CONTEXTUAL_PRECISION, CONTEXTUAL_RECALL, CONTEXTUAL_RELEVANCE |
+| Customizable prompt for response evaluation | ✅, with ASPECT_CRITIQUE metric | ❌ |
+| Explanations of scores | ❌ | ✅ |
+| Monitoring dashboard | ❌ | ❌ |
+
+:::info[Framework Documentation]
+
+You can find more information about the metrics in the documentation of the respective evaluation frameworks:
+
+- Ragas metrics: https://docs.ragas.io/en/latest/concepts/metrics/index.html
+- DeepEval metrics: https://docs.confident-ai.com/docs/metrics-introduction
+:::
+
+## Additional References
+
+:notebook: Tutorial: [Evaluating RAG Pipelines](https://haystack.deepset.ai/tutorials/35_evaluating_rag_pipelines)

docs-website/versioned_docs/version-2.22/optimization/evaluation/statistical-evaluation.mdx

@@ -0,0 +1,49 @@
+---
+title: "Statistical Evaluation"
+id: statistical-evaluation
+slug: "/statistical-evaluation"
+description: "Haystack supports various statistical evaluation metrics. This page explains what statistical evaluation is and discusses the various options available within Haystack."
+---
+
+# Statistical Evaluation
+
+Haystack supports various statistical evaluation metrics. This page explains what statistical evaluation is and discusses the various options available within Haystack.
+
+## Introduction
+
+Statistical evaluation in Haystack compares ground truth labels with pipeline predictions, typically using metrics such as precision or recall. It's often used to evaluate the Retriever component within Retrieval-Augmented Generative (RAG) pipelines, but this methodology can be adapted for any pipeline if ground truth labels of relevant documents are available.
+
+When evaluating answers, such as those predicted by an extractive question answering pipeline, the ground truth labels of expected answers are compared to the pipeline's predictions.
+
+For assessing answers generated by LLMs with one of Haystack’s Generator components, we recommend model-based evaluation instead. It can incorporate measures of semantic similarity or coherence and is better suited to evaluate predictions that might differ in wording from the ground truth labels.
+
+## Statistical Evaluation Pipelines in Haystack
+
+There are two ways of performing model-based evaluation in Haystack, both of which leverage [pipelines](../../concepts/pipelines.mdx) and [Evaluator](../../pipeline-components/evaluators.mdx) components:
+
+- You can create and run an evaluation pipeline independently. This means you’ll have to provide the required inputs to the evaluation pipeline manually. We recommend this way because the separation of your RAG pipeline and your evaluation pipeline allows you to store the results of your RAG pipeline and try out different evaluation metrics afterward without needing to re-run your pipeline every time.
+- As another option, you can add an Evaluator to the end of a RAG pipeline. This means you run both a RAG pipeline and evaluation on top of it in a single `pipeline.run()`  call.
+
+## Statistical Evaluation of Retrieved Documents
+
+### [DocumentRecallEvaluator](../../pipeline-components/evaluators/documentrecallevaluator.mdx)
+
+Recall measures how often the correct document was among the retrieved documents over a set of queries. For a single query, the output is binary: either the correct document is contained in the selection, or it is not. Over the entire dataset, the recall score amounts to a number between zero (no query retrieved the right document) and one (all queries retrieved the right documents).
+
+In some scenarios, there can be multiple correct documents for one query. The metric `recall_single_hit` considers whether at least one of the correct documents is retrieved, whereas `recall_multi_hit` takes into account how many of the multiple correct documents for one query are retrieved.
+
+Note that recall is affected by the number of documents that the Retriever returns. If the Retriever returns few documents, it means that it is difficult to retrieve the correct documents. Make sure to set the Retriever's `top_k` to an appropriate value in the pipeline that you're evaluating.
+
+### [DocumentMRREvaluator](../../pipeline-components/evaluators/documentmrrevaluator.mdx) (Mean Reciprocal Rank)
+
+In contrast to the recall metric, mean reciprocal rank takes the position of the top correctly retrieved document (the “rank”) into account. It does this to account for the fact that a query elicits multiple responses of varying relevance. Like recall, MRR can be a value between zero (no matches) and one (the system retrieved a correct document for all queries as the top result). For more details, check out [Mean Reciprocal Rank wiki page](https://en.wikipedia.org/wiki/Mean_reciprocal_rank).
+
+### [DocumentMAPEvaluator](../../pipeline-components/evaluators/documentmapevaluator.mdx) (Mean Average Precision)
+
+Mean average precision is similar to mean reciprocal rank but takes into account the position of every correctly retrieved document. Like MRR, mAP can be a value between zero (no matches) and one (the system retrieved correct documents for all top results). mAP is particularly useful in cases where there is more than one correct answer to be retrieved. For more details, check out [Mean Average Precision wiki page](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision).
+
+## Statistical Evaluation of Extracted or Generated Answers
+
+### [AnswerExactMatchEvaluator](../../pipeline-components/evaluators/answerexactmatchevaluator.mdx)
+
+Exact match measures the proportion of cases where the predicted Answer is identical to the correct Answer. For example, for the annotated question-answer pair “What is Haystack?" + "A question answering library in Python”, even a predicted answer like “A Python question answering library” would yield a zero score because it does not match the expected answer 100%.