chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
+93
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: "AnswerExactMatchEvaluator"
|
||||
id: answerexactmatchevaluator
|
||||
slug: "/answerexactmatchevaluator"
|
||||
description: "The `AnswerExactMatchEvaluator` evaluates answers predicted by Haystack pipelines using ground truth labels. It checks character by character whether a predicted answer exactly matches the ground truth answer. This metric is called the exact match."
|
||||
---
|
||||
|
||||
# AnswerExactMatchEvaluator
|
||||
|
||||
The `AnswerExactMatchEvaluator` evaluates answers predicted by Haystack pipelines using ground truth labels. It checks character by character whether a predicted answer exactly matches the ground truth answer. This metric is called the exact match.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `ground_truth_answers`: A list of strings containing the ground truth answers <br /> <br />`predicted_answers`: A list of strings containing the predicted answers to be evaluated |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 representing the proportion of questions in which any predicted answer matched the ground truth answers <br /> <br />- `individual_scores`: A list of 0s and 1s, where 1 means that the predicted answer matched one of the ground truths |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/answer_exact_match.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `AnswerExactMatchEvaluator` component to evaluate answers predicted by a Haystack pipeline, such as an extractive question answering pipeline, against ground truth labels. As the `AnswerExactMatchEvaluator` checks whether a predicted answer exactly matches the ground truth answer. It is not suited to evaluate answers generated by LLMs, for example, in a RAG pipeline. Use `FaithfulnessEvaluator` or `SASEvaluator` instead.
|
||||
|
||||
To initialize an `AnswerExactMatchEvaluator`, there are no parameters required.
|
||||
|
||||
Note that only _one_ predicted answer is compared to _one_ ground truth answer at a time. The component does not support multiple ground truth answers for the same question or multiple answers predicted for the same question.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example of using an `AnswerExactMatchEvaluator` component to evaluate two answers and compare them to ground truth answers.
|
||||
|
||||
```python
|
||||
from haystack.components.evaluators import AnswerExactMatchEvaluator
|
||||
|
||||
evaluator = AnswerExactMatchEvaluator()
|
||||
result = evaluator.run(
|
||||
ground_truth_answers=["Berlin", "Paris"],
|
||||
predicted_answers=["Berlin", "Lyon"],
|
||||
)
|
||||
|
||||
print(result["individual_scores"])
|
||||
## [1, 0]
|
||||
print(result["score"])
|
||||
## 0.5
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use an `AnswerExactMatchEvaluator` and a `SASEvaluator` in a pipeline to evaluate two answers and compare them to ground truth answers. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.evaluators import AnswerExactMatchEvaluator
|
||||
from haystack.components.evaluators import SASEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
em_evaluator = AnswerExactMatchEvaluator()
|
||||
sas_evaluator = SASEvaluator()
|
||||
pipeline.add_component("em_evaluator", em_evaluator)
|
||||
pipeline.add_component("sas_evaluator", sas_evaluator)
|
||||
|
||||
ground_truth_answers = ["Berlin", "Paris"]
|
||||
predicted_answers = ["Berlin", "Lyon"]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"em_evaluator": {
|
||||
"ground_truth_answers": ground_truth_answers,
|
||||
"predicted_answers": predicted_answers,
|
||||
},
|
||||
"sas_evaluator": {
|
||||
"ground_truth_answers": ground_truth_answers,
|
||||
"predicted_answers": predicted_answers,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1, 0]
|
||||
## [array([[0.99999994]], dtype=float32), array([[0.51747656]], dtype=float32)]
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 0.5
|
||||
## 0.7587383
|
||||
```
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: "ContextRelevanceEvaluator"
|
||||
id: contextrelevanceevaluator
|
||||
slug: "/contextrelevanceevaluator"
|
||||
description: "The `ContextRelevanceEvaluator` uses an LLM to evaluate whether contexts are relevant to a question. It does not require ground truth labels."
|
||||
---
|
||||
|
||||
# ContextRelevanceEvaluator
|
||||
|
||||
The `ContextRelevanceEvaluator` uses an LLM to evaluate whether contexts are relevant to a question. It does not require ground truth labels.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `questions`: A list of questions <br /> <br />`contexts`: A list of a list of contexts, which are the contents of documents. This accounts for one list of contexts per question. |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 that represents the mean average precision <br /> <br />- `individual_scores`: A list of the individual average precision scores ranging from 0.0 to 1.0 for each input pair of a list of retrieved documents and a list of ground truth documents <br /> <br />- `results`: A list of dictionaries with keys `statements` and `statement_scores`. They contain the statements extracted by an LLM from each context and the corresponding context relevance scores per statement, which are either 0 or 1. |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/context_relevance.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `ContextRelevanceEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, without ground truth labels. The component breaks up the context into multiple statements and checks whether each statement is relevant for answering a question. The final score for the context relevance is a number from 0.0 to 1.0 and represents the proportion of statements that are relevant to the provided question.
|
||||
|
||||
### Parameters
|
||||
|
||||
The default model for this Evaluator is `gpt-4o-mini`. You can override the model using the `chat_generator` parameter during initialization. This needs to be a Chat Generator instance configured to return a JSON object. For example, when using the [`OpenAIChatGenerator`](../generators/openaichatgenerator.mdx), you should pass `{"response_format": {"type": "json_object"}}` in its `generation_kwargs`.
|
||||
|
||||
If you are not initializing the Evaluator with your own Chat Generator other than OpenAI, a valid OpenAI API key must be set as an `OPENAI_API_KEY` environment variable. For details, see our [documentation page on secret management](../../concepts/secret-management.mdx).
|
||||
|
||||
Two optional initialization parameters are:
|
||||
|
||||
- `raise_on_failure`: If True, raise an exception on an unsuccessful API call.
|
||||
- `progress_bar`: Whether to show a progress bar during the evaluation.
|
||||
|
||||
`ContextRelevanceEvaluator` has an optional `examples` parameter that can be used to pass few-shot examples conforming to the expected input and output format of `ContextRelevanceEvaluator`. These examples are included in the prompt that is sent to the LLM. Examples, therefore, increase the number of tokens of the prompt and make each request more costly. Adding examples is helpful if you want to improve the quality of the evaluation at the cost of more tokens.
|
||||
|
||||
Each example must be a dictionary with keys `inputs` and `outputs`.
|
||||
`inputs` must be a dictionary with keys `questions` and `contexts`.
|
||||
`outputs` must be a dictionary with `statements` and `statement_scores`.
|
||||
Here is the expected format:
|
||||
|
||||
```python
|
||||
[
|
||||
{
|
||||
"inputs": {
|
||||
"questions": "What is the capital of Italy?",
|
||||
"contexts": ["Rome is the capital of Italy."],
|
||||
},
|
||||
"outputs": {
|
||||
"statements": [
|
||||
"Rome is the capital of Italy.",
|
||||
"Rome has more than 4 million inhabitants.",
|
||||
],
|
||||
"statement_scores": [1, 0],
|
||||
},
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use a `ContextRelevanceEvaluator` component to evaluate a response generated based on a provided question and context. The `ContextRelevanceEvaluator` returns a score of 1.0 because it detects one statement in the context, which is relevant to the question.
|
||||
|
||||
```python
|
||||
from haystack.components.evaluators import ContextRelevanceEvaluator
|
||||
|
||||
questions = ["Who created the Python language?"]
|
||||
contexts = [
|
||||
[
|
||||
"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
|
||||
],
|
||||
]
|
||||
|
||||
evaluator = ContextRelevanceEvaluator()
|
||||
result = evaluator.run(questions=questions, contexts=contexts)
|
||||
print(result["score"])
|
||||
## 1.0
|
||||
print(result["individual_scores"])
|
||||
## [1.0]
|
||||
print(result["results"])
|
||||
## [{'statements': ['Python, created by Guido van Rossum in the late 1980s.'], 'statement_scores': [1], 'score': 1.0}]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use a `FaithfulnessEvaluator` and a `ContextRelevanceEvaluator` in a pipeline to evaluate responses and contexts (the content of documents) received by a RAG pipeline based on provided questions. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.evaluators import (
|
||||
ContextRelevanceEvaluator,
|
||||
FaithfulnessEvaluator,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
context_relevance_evaluator = ContextRelevanceEvaluator()
|
||||
faithfulness_evaluator = FaithfulnessEvaluator()
|
||||
pipeline.add_component("context_relevance_evaluator", context_relevance_evaluator)
|
||||
pipeline.add_component("faithfulness_evaluator", faithfulness_evaluator)
|
||||
|
||||
questions = ["Who created the Python language?"]
|
||||
contexts = [
|
||||
[
|
||||
"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
|
||||
],
|
||||
]
|
||||
responses = [
|
||||
"Python is a high-level general-purpose programming language that was created by George Lucas.",
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"context_relevance_evaluator": {"questions": questions, "contexts": contexts},
|
||||
"faithfulness_evaluator": {
|
||||
"questions": questions,
|
||||
"contexts": contexts,
|
||||
"responses": responses,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1.0]
|
||||
## [0.5]
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 1.0
|
||||
## 0.5
|
||||
```
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: "DeepEvalEvaluator"
|
||||
id: deepevalevaluator
|
||||
slug: "/deepevalevaluator"
|
||||
description: "The DeepEvalEvaluator evaluates Haystack pipelines using LLM-based metrics. It supports metrics like answer relevancy, faithfulness, contextual relevance, and more."
|
||||
---
|
||||
|
||||
# DeepEvalEvaluator
|
||||
|
||||
The DeepEvalEvaluator evaluates Haystack pipelines using LLM-based metrics. It supports metrics like answer relevancy, faithfulness, contextual relevance, and more.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `metric`: One of the DeepEval metrics to use for evaluation |
|
||||
| **Mandatory run variables** | `**inputs`: A keyword arguments dictionary containing the expected inputs. The expected inputs will change based on the metric you are evaluating. See below for more details. |
|
||||
| **Output variables** | `results`: A nested list of metric results. There can be one or more results, depending on the metric. Each result is a dictionary containing: <br /> <br />- `name` - The name of the metric <br />- `score` - The score of the metric <br />- `explanation` - An optional explanation of the score |
|
||||
| **API reference** | [DeepEval](/reference/integrations-deepeval) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/deepeval |
|
||||
|
||||
</div>
|
||||
|
||||
DeepEval is an evaluation framework that provides a number of LLM-based evaluation metrics. You can use the `DeepEvalEvaluator` component to evaluate a Haystack pipeline, such as a retrieval-augmented generated pipeline, against one of the metrics provided by DeepEval.
|
||||
|
||||
## Supported Metrics
|
||||
|
||||
DeepEval supports a number of metrics, which we expose through the [DeepEval metric enumeration.](/reference/integrations-deepeval#deepevalmetric) [`DeepEvalEvaluator`](/reference/integrations-deepeval#deepevalevaluator) in Haystack supports the metrics listed below with the expected `metric_params` while initializing the Evaluator. Many metrics use OpenAI models and require you to set an environment variable `OPENAI_API_KEY`. For a complete guide on these metrics, visit the [DeepEval documentation](https://docs.confident-ai.com/docs/getting-started).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `metric`: One of the DeepEval metrics to use for evaluation |
|
||||
| **Mandatory run variables** | “\*\*inputs”: A keyword arguments dictionary containing the expected inputs. The expected inputs will change based on the metric you are evaluating. See below for more details. |
|
||||
| **Output variables** | `results`: A nested list of metric results. There can be one or more results, depending on the metric. Each result is a dictionary containing: <br /> <br />- `name` - The name of the metric <br />- `score` - The score of the metric <br />- `explanation` - An optional explanation of the score |
|
||||
| **API reference** | [DeepEval](/reference/integrations-deepeval) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/deepeval |
|
||||
|
||||
</div>
|
||||
|
||||
## Parameters Overview
|
||||
|
||||
To initialize a `DeepEvalEvaluator`, you need to provide the following parameters :
|
||||
|
||||
- `metric`: A `DeepEvalMetric`.
|
||||
- `metric_params`: Optionally, if the metric calls for any additional parameters, you should provide them here.
|
||||
|
||||
## Usage
|
||||
|
||||
To use the `DeepEvalEvaluator`, you first need to install the integration:
|
||||
|
||||
```bash
|
||||
pip install deepeval-haystack
|
||||
```
|
||||
|
||||
To use the `DeepEvalEvaluator` you need to follow these steps:
|
||||
|
||||
1. Initialize the `DeepEvalEvaluator` while providing the correct `metric_params` for the metric you are using.
|
||||
2. Run the `DeepEvalEvaluator` on its own or in a pipeline by providing the expected input for the metric you are using.
|
||||
|
||||
### Examples
|
||||
|
||||
**Evaluate Faithfulness**
|
||||
|
||||
To create a faithfulness evaluation pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.evaluators.deepeval import (
|
||||
DeepEvalEvaluator,
|
||||
DeepEvalMetric,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
evaluator = DeepEvalEvaluator(
|
||||
metric=DeepEvalMetric.FAITHFULNESS,
|
||||
metric_params={"model": "gpt-4"},
|
||||
)
|
||||
pipeline.add_component("evaluator", evaluator)
|
||||
```
|
||||
|
||||
To run the evaluation pipeline, you should have the _expected inputs_ for the metric ready at hand. This metric expects a list of `questions` and `contexts`. These should come from the results of the pipeline you want to evaluate.
|
||||
|
||||
```python
|
||||
results = pipeline.run(
|
||||
{
|
||||
"evaluator": {
|
||||
"questions": [
|
||||
"When was the Rhodes Statue built?",
|
||||
"Where is the Pyramid of Giza?",
|
||||
],
|
||||
"contexts": [["Context for question 1"], ["Context for question 2"]],
|
||||
"responses": ["Response for question 1", "response for question 2"],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [RAG Pipeline Evaluation Using DeepEval](https://haystack.deepset.ai/cookbook/rag_eval_deep_eval)
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "DocumentMAPEvaluator"
|
||||
id: documentmapevaluator
|
||||
slug: "/documentmapevaluator"
|
||||
description: "The `DocumentMAPEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks to what extent the list of retrieved documents contains only relevant documents as specified in the ground truth labels or also non-relevant documents. This metric is called mean average precision (MAP)."
|
||||
---
|
||||
|
||||
# DocumentMAPEvaluator
|
||||
|
||||
The `DocumentMAPEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks to what extent the list of retrieved documents contains only relevant documents as specified in the ground truth labels or also non-relevant documents. This metric is called mean average precision (MAP).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `ground_truth_documents`: A list of a list of ground truth documents. This accounts for one list of ground truth documents per question. <br /> <br />`retrieved_documents`: A list of a list of retrieved documents. This accounts for one list of retrieved documents per question. |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 that represents the mean average precision <br /> <br />- `individual_scores`: A list of the individual average precision scores ranging from 0.0 to 1.0 for each input pair of a list of retrieved documents and a list of ground truth documents |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/document_map.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `DocumentMAPEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, against ground truth labels. A higher mean average precision is better, indicating that the list of retrieved documents contains many relevant documents and only a few non-relevant documents or none at all.
|
||||
|
||||
To initialize a `DocumentMAPEvaluator`, there are no parameters required.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use a `DocumentMAPEvaluator` component to evaluate documents retrieved for two queries. For the first query, there is one ground truth document and one retrieved document. For the second query, there are two ground truth documents and three retrieved documents.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.evaluators import DocumentMAPEvaluator
|
||||
|
||||
evaluator = DocumentMAPEvaluator()
|
||||
result = evaluator.run(
|
||||
ground_truth_documents=[
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
],
|
||||
retrieved_documents=[
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
],
|
||||
)
|
||||
print(result["individual_scores"])
|
||||
## [1.0, 0.8333333333333333]
|
||||
print(result["score"])
|
||||
## 0.9166666666666666
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use a `DocumentMAPEvaluator` and a `DocumentMRREvaluator` in a pipeline to evaluate two answers and compare them to ground truth answers. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.evaluators import DocumentMRREvaluator, DocumentMAPEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
mrr_evaluator = DocumentMRREvaluator()
|
||||
map_evaluator = DocumentMAPEvaluator()
|
||||
pipeline.add_component("mrr_evaluator", mrr_evaluator)
|
||||
pipeline.add_component("map_evaluator", map_evaluator)
|
||||
|
||||
ground_truth_documents = [
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
]
|
||||
retrieved_documents = [
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"mrr_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
"map_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1.0, 1.0]
|
||||
## [1.0, 0.8333333333333333]
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 1.0
|
||||
## 0.9166666666666666
|
||||
```
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "DocumentMRREvaluator"
|
||||
id: documentmrrevaluator
|
||||
slug: "/documentmrrevaluator"
|
||||
description: "The `DocumentMRREvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks at what rank ground truth documents appear in the list of retrieved documents. This metric is called mean reciprocal rank (MRR)."
|
||||
---
|
||||
|
||||
# DocumentMRREvaluator
|
||||
|
||||
The `DocumentMRREvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks at what rank ground truth documents appear in the list of retrieved documents. This metric is called mean reciprocal rank (MRR).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `ground_truth_documents`: A list containing another list of ground truth documents. This accounts for one list of ground truth documents per question. <br /> <br />`retrieved_documents`: A list containing another list of retrieved documents. This accounts for one list of retrieved documents per question. |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 that represents the mean reciprocal rank <br /> <br />- `individual_scores`: A list of the individual reciprocal ranks ranging from 0.0 to 1.0 for each input pair of a list of retrieved documents and a list of ground truth documents |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/document_mrr.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `DocumentMRREvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, against ground truth labels. A higher mean reciprocal rank is better and indicates that relevant documents appear at an earlier position in the list of retrieved documents.
|
||||
|
||||
To initialize a `DocumentMRREvaluator`, there are no parameters required.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use a `DocumentMRREvaluator` component to evaluate documents retrieved for two queries. For the first query, there is one ground truth document and one retrieved document. For the second query, there are two ground truth documents and three retrieved documents.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.evaluators import DocumentMRREvaluator
|
||||
|
||||
evaluator = DocumentMRREvaluator()
|
||||
result = evaluator.run(
|
||||
ground_truth_documents=[
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
],
|
||||
retrieved_documents=[
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
],
|
||||
)
|
||||
print(result["individual_scores"])
|
||||
## [1.0, 1.0]
|
||||
print(result["score"])
|
||||
## 1.0
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use a `DocumentRecallEvaluator` and a `DocumentMRREvaluator` in a pipeline to evaluate two answers and compare them to ground truth answers. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.evaluators import DocumentMRREvaluator, DocumentRecallEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
mrr_evaluator = DocumentMRREvaluator()
|
||||
recall_evaluator = DocumentRecallEvaluator()
|
||||
pipeline.add_component("mrr_evaluator", mrr_evaluator)
|
||||
pipeline.add_component("recall_evaluator", recall_evaluator)
|
||||
|
||||
ground_truth_documents = [
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
]
|
||||
retrieved_documents = [
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"mrr_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
"recall_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1.0, 1.0]
|
||||
## [1.0, 1.0]
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 1.0
|
||||
## 1.0
|
||||
```
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "DocumentNDCGEvaluator"
|
||||
id: documentndcgevaluator
|
||||
slug: "/documentndcgevaluator"
|
||||
description: "The `DocumentNDCGEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks at what rank ground truth documents appear in the list of retrieved documents. This metric is called normalized discounted cumulative gain (NDCG)."
|
||||
---
|
||||
|
||||
# DocumentNDCGEvaluator
|
||||
|
||||
The `DocumentNDCGEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks at what rank ground truth documents appear in the list of retrieved documents. This metric is called normalized discounted cumulative gain (NDCG).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `ground_truth_documents`: A list containing another list of ground truth documents, one list per question <br /> <br />`retrieved_documents`: A list containing another list of retrieved documents, one list per question |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 that represents the NDCG <br /> <br />- `individual_scores`: A list of individual NDCG values ranging from 0.0 to 1.0 for each input pair of a list of retrieved documents and a list of ground truth documents |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/document_ndcg.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `DocumentNDCGEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, against ground truth labels. A higher NDCG is better and indicates that relevant documents appear at an earlier position in the list of retrieved documents.
|
||||
|
||||
If the ground truth documents have scores, a higher NDCG indicates that documents with a higher score appear at an earlier position in the list of retrieved documents. If the ground truth documents have no scores, binary relevance is assumed, meaning that all ground truth documents are equally relevant, and the order in which they are in the list of retrieved documents does not matter for the NDCG.
|
||||
|
||||
No parameters are required to initialize a `DocumentNDCGEvaluator`.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use the `DocumentNDCGEvaluator` to evaluate documents retrieved for a query. There are two ground truth documents and three retrieved documents. All ground truth documents are retrieved, but one non-relevant document is ranked higher than one of the ground truth documents, which lowers the NDCG score.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.evaluators import DocumentNDCGEvaluator
|
||||
|
||||
evaluator = DocumentNDCGEvaluator()
|
||||
result = evaluator.run(
|
||||
ground_truth_documents=[
|
||||
[Document(content="France", score=1.0), Document(content="Paris", score=0.5)],
|
||||
],
|
||||
retrieved_documents=[
|
||||
[
|
||||
Document(content="France"),
|
||||
Document(content="Germany"),
|
||||
Document(content="Paris"),
|
||||
],
|
||||
],
|
||||
)
|
||||
print(result["individual_scores"])
|
||||
## [0.8869]
|
||||
print(result["score"])
|
||||
## 0.8869
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of using a `DocumentNDCGEvaluator` and `DocumentMRREvaluator` in a pipeline to evaluate retrieved documents and compare them to ground truth documents. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.evaluators import DocumentMRREvaluator, DocumentNDCGEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("ndcg_evaluator", DocumentNDCGEvaluator())
|
||||
pipeline.add_component("mrr_evaluator", DocumentMRREvaluator())
|
||||
|
||||
ground_truth_documents = [
|
||||
[Document(content="France", score=1.0), Document(content="Paris", score=0.5)],
|
||||
]
|
||||
retrieved_documents = [
|
||||
[
|
||||
Document(content="France"),
|
||||
Document(content="Germany"),
|
||||
Document(content="Paris"),
|
||||
],
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"ndcg_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
"mrr_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 0.9502
|
||||
## 1.0
|
||||
```
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
---
|
||||
title: "DocumentRecallEvaluator"
|
||||
id: documentrecallevaluator
|
||||
slug: "/documentrecallevaluator"
|
||||
description: "The `DocumentRecallEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks how many of the ground truth documents were retrieved. This metric is called recall."
|
||||
---
|
||||
|
||||
# DocumentRecallEvaluator
|
||||
|
||||
The `DocumentRecallEvaluator` evaluates documents retrieved by Haystack pipelines using ground truth labels. It checks how many of the ground truth documents were retrieved. This metric is called recall.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `ground_truth_documents`: A list of a list of ground truth documents. This accounts for one list of ground truth documents per question. <br /> <br />`retrieved_documents`: A list of a list of retrieved documents. This accounts for one list of retrieved documents per question. |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 that represents the mean recall score over all inputs <br /> <br />- `individual_scores`: A list of the individual recall scores ranging from 0.0 to 1.0 of each input pair of a list of retrieved documents and a list of ground truth documents. If the mode is set to single_hit, each individual score is either 0 or 1. |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/document_recall.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `DocumentRecallEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG Pipeline, against ground truth labels.
|
||||
|
||||
When initializing a `DocumentRecallEvaluator`, you can set the `mode` parameter to
|
||||
`RecallMode.SINGLE_HIT` or `RecallMode.MULTI_HIT`. By default, `RecallMode.SINGLE_HIT` is used.
|
||||
|
||||
`RecallMode.SINGLE_HIT` means that _any_ of the ground truth documents need to be retrieved to count as a correct retrieval with a recall score of 1. A single retrieved document can achieve the full score.
|
||||
|
||||
`RecallMode.MULTI_HIT` means that _all_ of the ground truth documents need to be retrieved to count as a correct retrieval with a recall score of 1. The number of retrieved documents must be at least the number of ground truth documents to achieve the full score.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use a `DocumentRecallEvaluator` component to evaluate documents retrieved for two queries. For the first query, there is one ground truth document and one retrieved document. For the second query, there are two ground truth documents and three retrieved documents.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.evaluators import DocumentRecallEvaluator
|
||||
|
||||
evaluator = DocumentRecallEvaluator()
|
||||
result = evaluator.run(
|
||||
ground_truth_documents=[
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
],
|
||||
retrieved_documents=[
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
],
|
||||
)
|
||||
print(result["individual_scores"])
|
||||
## [1.0, 1.0]
|
||||
print(result["score"])
|
||||
## 1.0
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use a `DocumentRecallEvaluator` and a `DocumentMRREvaluator` in a pipeline to evaluate two answers and compare them to ground truth answers. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.evaluators import DocumentMRREvaluator, DocumentRecallEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
mrr_evaluator = DocumentMRREvaluator()
|
||||
recall_evaluator = DocumentRecallEvaluator()
|
||||
pipeline.add_component("mrr_evaluator", mrr_evaluator)
|
||||
pipeline.add_component("recall_evaluator", recall_evaluator)
|
||||
|
||||
ground_truth_documents = [
|
||||
[Document(content="France")],
|
||||
[Document(content="9th century"), Document(content="9th")],
|
||||
]
|
||||
retrieved_documents = [
|
||||
[Document(content="France")],
|
||||
[
|
||||
Document(content="9th century"),
|
||||
Document(content="10th century"),
|
||||
Document(content="9th"),
|
||||
],
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"mrr_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
"recall_evaluator": {
|
||||
"ground_truth_documents": ground_truth_documents,
|
||||
"retrieved_documents": retrieved_documents,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1.0, 1.0]
|
||||
## [1.0, 1.0]
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 1.0
|
||||
## 1.0
|
||||
```
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-evaluators
|
||||
slug: "/external-integrations-evaluators"
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [Flow Judge](https://haystack.deepset.ai/integrations/flow-judge) | Evaluate Haystack pipelines using Flow Judge model. |
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "FaithfulnessEvaluator"
|
||||
id: faithfulnessevaluator
|
||||
slug: "/faithfulnessevaluator"
|
||||
description: "The `FaithfulnessEvaluator` uses an LLM to evaluate whether a generated answer can be inferred from the provided contexts. It does not require ground truth labels. This metric is called faithfulness, sometimes also referred to as groundedness or hallucination."
|
||||
---
|
||||
|
||||
# FaithfulnessEvaluator
|
||||
|
||||
The `FaithfulnessEvaluator` uses an LLM to evaluate whether a generated answer can be inferred from the provided contexts. It does not require ground truth labels. This metric is called faithfulness, sometimes also referred to as groundedness or hallucination.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory run variables** | `questions`: A list of questions <br /> <br />`contexts`: A list of a list of contexts, which are the contents of documents. This accounts for one list of contexts per question. <br /> <br />`predicted_answers`: A list of predicted answers, for example, the outputs of a Generator in a RAG pipeline |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />- `score`: A number from 0.0 to 1.0 that represents the average faithfulness score across all questions <br /> <br />- `individual_scores`: A list of the individual faithfulness scores ranging from 0.0 to 1.0 for each input triple of a question, a list of contexts, and a predicted answer. <br /> <br />- `results`: A list of dictionaries with `statements` and `statement_scores` keys. They contain the statements extracted by an LLM from each predicted answer and the corresponding faithfulness scores per statement, which are either 0 or 1. |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/faithfulness.py |
|
||||
|
||||
</div>
|
||||
|
||||
You can use the `FaithfulnessEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, without ground truth labels. The component splits the generated answer into statements and checks each of them against the provided contexts with an LLM. A higher faithfulness score is better, and it indicates that a larger number of statements in the generated answers can be inferred from the contexts. The faithfulness score can be used to better understand how often and when the Generator in a RAG pipeline hallucinates.
|
||||
|
||||
### Parameters
|
||||
|
||||
The default model for this Evaluator is `gpt-4o-mini`. You can override the model using the `chat_generator` parameter during initialization. This needs to be a Chat Generator instance configured to return a JSON object. For example, when using the [`OpenAIChatGenerator`](../generators/openaichatgenerator.mdx), you should pass `{"response_format": {"type": "json_object"}}` in its `generation_kwargs`.
|
||||
|
||||
If you are not initializing the Evaluator with your own Chat Generator other than OpenAI, a valid OpenAI API key must be set as an `OPENAI_API_KEY` environment variable. For details, see our [documentation page on secret management](../../concepts/secret-management.mdx).
|
||||
|
||||
Two other optional initialization parameters are:
|
||||
|
||||
- `raise_on_failure`: If True, raise an exception on an unsuccessful API call.
|
||||
- `progress_bar`: Whether to show a progress bar during the evaluation.
|
||||
|
||||
`FaithfulnessEvaluator` has an optional `examples` parameter that can be used to pass few-shot examples conforming to the expected input and output format of `FaithfulnessEvaluator`. These examples are included in the prompt that is sent to the LLM. Examples, therefore, increase the number of tokens of the prompt and make each request more costly. Adding examples is helpful if you want to improve the quality of the evaluation at the cost of more tokens.
|
||||
|
||||
Each example must be a dictionary with keys `inputs` and `outputs`.
|
||||
`inputs` must be a dictionary with keys `questions`, `contexts`, and `predicted_answers`.
|
||||
`outputs` must be a dictionary with `statements` and `statement_scores`.
|
||||
Here is the expected format:
|
||||
|
||||
```python
|
||||
[
|
||||
{
|
||||
"inputs": {
|
||||
"questions": "What is the capital of Italy?",
|
||||
"contexts": ["Rome is the capital of Italy."],
|
||||
"predicted_answers": "Rome is the capital of Italy with more than 4 million inhabitants.",
|
||||
},
|
||||
"outputs": {
|
||||
"statements": [
|
||||
"Rome is the capital of Italy.",
|
||||
"Rome has more than 4 million inhabitants.",
|
||||
],
|
||||
"statement_scores": [1, 0],
|
||||
},
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example of using a `FaithfulnessEvaluator` component to evaluate a predicted answer generated based on a provided question and context. The `FaithfulnessEvaluator` returns a score of 0.5 because it detects two statements in the answer, of which only one is correct.
|
||||
|
||||
```python
|
||||
from haystack.components.evaluators import FaithfulnessEvaluator
|
||||
|
||||
questions = ["Who created the Python language?"]
|
||||
contexts = [
|
||||
[
|
||||
"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
|
||||
],
|
||||
]
|
||||
predicted_answers = [
|
||||
"Python is a high-level general-purpose programming language that was created by George Lucas.",
|
||||
]
|
||||
evaluator = FaithfulnessEvaluator()
|
||||
result = evaluator.run(
|
||||
questions=questions,
|
||||
contexts=contexts,
|
||||
predicted_answers=predicted_answers,
|
||||
)
|
||||
|
||||
print(result["individual_scores"])
|
||||
## [0.5]
|
||||
print(result["score"])
|
||||
## 0.5
|
||||
print(result["results"])
|
||||
## [{'statements': ['Python is a high-level general-purpose programming language.',
|
||||
## 'Python was created by George Lucas.'], 'statement_scores': [1, 0], 'score': 0.5}]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use a `FaithfulnessEvaluator` and a `ContextRelevanceEvaluator` in a pipeline to evaluate predicted answers and contexts (the content of documents) received by a RAG pipeline based on provided questions. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.evaluators import (
|
||||
ContextRelevanceEvaluator,
|
||||
FaithfulnessEvaluator,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
context_relevance_evaluator = ContextRelevanceEvaluator()
|
||||
faithfulness_evaluator = FaithfulnessEvaluator()
|
||||
pipeline.add_component("context_relevance_evaluator", context_relevance_evaluator)
|
||||
pipeline.add_component("faithfulness_evaluator", faithfulness_evaluator)
|
||||
|
||||
questions = ["Who created the Python language?"]
|
||||
contexts = [
|
||||
[
|
||||
"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
|
||||
],
|
||||
]
|
||||
predicted_answers = [
|
||||
"Python is a high-level general-purpose programming language that was created by George Lucas.",
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"context_relevance_evaluator": {"questions": questions, "contexts": contexts},
|
||||
"faithfulness_evaluator": {
|
||||
"questions": questions,
|
||||
"contexts": contexts,
|
||||
"predicted_answers": predicted_answers,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## ...
|
||||
## [0.5]
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
##
|
||||
## 0.5
|
||||
```
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
---
|
||||
title: "LLMEvaluator"
|
||||
id: llmevaluator
|
||||
slug: "/llmevaluator"
|
||||
description: "This Evaluator uses an LLM to evaluate inputs based on a prompt containing user-defined instructions and examples."
|
||||
---
|
||||
|
||||
# LLMEvaluator
|
||||
|
||||
This Evaluator uses an LLM to evaluate inputs based on a prompt containing user-defined instructions and examples.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `instructions`: The prompt instructions string <br /> <br />`inputs`: The expected inputs <br /> <br />`outputs`: The output names of the evaluation results <br /> <br />`examples`: Few-shot examples conforming to the input and output format |
|
||||
| **Mandatory run variables** | `inputs`: Defined by the user – for example, questions or responses |
|
||||
| **Output variables** | `results`: A dictionary containing keys defined by the user, such as score |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/llm_evaluator.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `LLMEvaluator` component can evaluate answers, documents, or any other outputs of a Haystack pipeline based on a user-defined aspect. The component combines the instructions, examples, and expected output names into one prompt. It is meant for calculating user-defined model-based evaluation metrics. If you are looking for pre-defined model-based evaluators that work out of the box, have a look at Haystack’s [`FaithfulnessEvaluator`](faithfulnessevaluator.mdx) and [`ContextRelevanceEvaluator`](contextrelevanceevaluator.mdx) components instead.
|
||||
|
||||
### Parameters
|
||||
|
||||
The default model for this Evaluator is `gpt-4o-mini`. You can override the model using the `chat_generator` parameter during initialization. This needs to be a Chat Generator instance configured to return a JSON object. For example, when using the [`OpenAIChatGenerator`](../generators/openaichatgenerator.mdx), you should pass `{"response_format": {"type": "json_object"}}` in its `generation_kwargs`.
|
||||
|
||||
If you are not initializing the Evaluator with your own Chat Generator other than OpenAI, a valid OpenAI API key must be set as an `OPENAI_API_KEY` environment variable. For details, see our [documentation page on secret management](../../concepts/secret-management.mdx).
|
||||
|
||||
`LLMEvaluator` requires six parameters for initialization:
|
||||
|
||||
- `instructions`: The prompt instructions to use for evaluation, such as a question about the inputs that the LLM can answer with _yes,_ _no_, or a score.
|
||||
- `inputs`: The inputs that the `LLMEvaluator` expects and that it evaluates. The inputs determine the incoming connections of the component. Each input is a tuple of an input name and input type. Input types must be lists. An example could be `[("responses", List[str])]`.
|
||||
- `outputs`: Output names of the evaluation results corresponding to keys in the output dictionary. An example could be a `["score"]`.
|
||||
- `examples`: Use this parameter to pass few-shot examples conforming to the expected input and output format. These examples are included in the prompt that is sent to the LLM. Examples increase the number of tokens of the prompt and make each request more costly. Adding more than one or two examples can be helpful if you want to improve the quality of the evaluation at the cost of more tokens.
|
||||
- `raise_on_failure`: If True (default), raise an exception on an unsuccessful API call.
|
||||
- `progress_bar`: Whether to show a progress bar during the evaluation. None is the default.
|
||||
|
||||
Each example must be a dictionary with keys `inputs` and `outputs`.
|
||||
`inputs` must be a dictionary with keys `questions` and `contexts`.
|
||||
`outputs` must be a dictionary with `statements` and `statement_scores`.
|
||||
|
||||
Here is the expected format:
|
||||
|
||||
```python
|
||||
[
|
||||
{
|
||||
"inputs": {
|
||||
"questions": "What is the capital of Italy?",
|
||||
"contexts": ["Rome is the capital of Italy."],
|
||||
},
|
||||
"outputs": {
|
||||
"statements": [
|
||||
"Rome is the capital of Italy.",
|
||||
"Rome has more than 4 million inhabitants.",
|
||||
],
|
||||
"statement_scores": [1, 0],
|
||||
},
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we use an `LLMEvaluator` component to evaluate a generated response. The aspect we evaluate is whether the response is problematic for children as defined in the instructions. The `LLMEvaluator` returns one binary score per input response with the result that both responses are not problematic.
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
from haystack.components.evaluators import LLMEvaluator
|
||||
|
||||
llm_evaluator = LLMEvaluator(
|
||||
instructions="Is this answer problematic for children?",
|
||||
inputs=[("responses", List[str])],
|
||||
outputs=["score"],
|
||||
examples=[
|
||||
{
|
||||
"inputs": {"responses": "Damn, this is straight outta hell!!!"},
|
||||
"outputs": {"score": 1},
|
||||
},
|
||||
{
|
||||
"inputs": {"responses": "Football is the most popular sport."},
|
||||
"outputs": {"score": 0},
|
||||
},
|
||||
],
|
||||
)
|
||||
responses = [
|
||||
"Football is the most popular sport with around 4 billion followers worldwide",
|
||||
"Python language was created by Guido van Rossum.",
|
||||
]
|
||||
results = llm_evaluator.run(responses=responses)
|
||||
print(results)
|
||||
## {'results': [{'score': 0}, {'score': 0}]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use an `LLMEvaluator` in a pipeline to evaluate a response.
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
from haystack import Pipeline
|
||||
from haystack.components.evaluators import LLMEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
llm_evaluator = LLMEvaluator(
|
||||
instructions="Is this answer problematic for children?",
|
||||
inputs=[("responses", List[str])],
|
||||
outputs=["score"],
|
||||
examples=[
|
||||
{
|
||||
"inputs": {"responses": "Damn, this is straight outta hell!!!"},
|
||||
"outputs": {"score": 1},
|
||||
},
|
||||
{
|
||||
"inputs": {"responses": "Football is the most popular sport."},
|
||||
"outputs": {"score": 0},
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
pipeline.add_component("llm_evaluator", llm_evaluator)
|
||||
|
||||
responses = [
|
||||
"Football is the most popular sport with around 4 billion followers worldwide",
|
||||
"Python language was created by Guido van Rossum.",
|
||||
]
|
||||
|
||||
result = pipeline.run({"llm_evaluator": {"responses": responses}})
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["results"])
|
||||
## [{'score': 0}, {'score': 0}]
|
||||
```
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
---
|
||||
title: "RagasEvaluator"
|
||||
id: ragasevaluator
|
||||
slug: "/ragasevaluator"
|
||||
description: "This component evaluates Haystack pipelines using LLM-based metrics. It supports metrics like context relevance, factual accuracy, response relevance, and more."
|
||||
---
|
||||
|
||||
# RagasEvaluator
|
||||
|
||||
This component evaluates Haystack pipelines using LLM-based metrics. It supports metrics like context relevance, factual accuracy, response relevance, and more.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `metric`: A Ragas metric to use for evaluation |
|
||||
| **Mandatory run variables** | `inputs`: A keyword arguments dictionary containing the expected inputs. The expected inputs will change based on the metric you are evaluating. See below for more details. |
|
||||
| **Output variables** | `results`: A nested list of metric results. There can be one or more results, depending on the metric. Each result is a dictionary containing: <br /> <br />- `name` - The name of the metric. <br /> <br />- `score` - The score of the metric. |
|
||||
| **API reference** | [Ragas](/reference/integrations-ragas) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ragas |
|
||||
|
||||
</div>
|
||||
|
||||
Ragas is an evaluation framework that provides a number of LLM-based evaluation metrics. You can use the `RagasEvaluator` component to evaluate a Haystack pipeline, such as a retrieval-augmented generative pipeline, against one of the metrics provided by Ragas.
|
||||
|
||||
## Supported Metrics
|
||||
|
||||
Ragas supports a number of metrics, which we expose through the Ragas metric enumeration. Below is the list of metrics supported by the `RagasEvaluator` in Haystack with the expected `metric_params` while initializing the evaluator. Many metrics use OpenAI models and require an environment variable `OPENAI_API_KEY` to be set. For a complete guide on these metrics, visit the [Ragas documentation](https://docs.ragas.io/).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `metric`: A Ragas metric to use for evaluation |
|
||||
| **Mandatory run variables** | `inputs`: A keyword arguments dictionary containing the expected inputs. The expected inputs will change based on the metric you are evaluating. See below for more details. |
|
||||
| **Output variables** | `results`: A nested list of metric results. There can be one or more results, depending on the metric. Each result is a dictionary containing: <br /> <br />- `name` - The name of the metric. <br /> <br />- `score` - The score of the metric. |
|
||||
| **API reference** | [Ragas](/reference/integrations-ragas) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ragas |
|
||||
|
||||
</div>
|
||||
|
||||
## Parameters Overview
|
||||
|
||||
To initialize a `RagasEvaluator`, you need to provide the following parameters :
|
||||
|
||||
- `metric`: A `RagasMetric`.
|
||||
- `metric_params`: Optionally, if the metric calls for any additional parameters, you should provide them here.
|
||||
|
||||
## Usage
|
||||
|
||||
To use the `RagasEvaluator`, you first need to install the integration:
|
||||
|
||||
```bash
|
||||
pip install ragas-haystack
|
||||
```
|
||||
|
||||
To use the `RagasEvaluator` you need to follow these steps:
|
||||
|
||||
1. Initialize the `RagasEvaluator` while providing the correct `metric_params` for the metric you are using.
|
||||
2. Run the `RagasEvaluator`, either on its own or in a pipeline, by providing the expected input for the metric you are using.
|
||||
|
||||
### Examples
|
||||
|
||||
#### Evaluate Context Relevance
|
||||
|
||||
To create a context-relevance evaluation pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.evaluators.ragas import (
|
||||
RagasEvaluator,
|
||||
RagasMetric,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
evaluator = RagasEvaluator(
|
||||
metric=RagasMetric.ANSWER_RELEVANCY,
|
||||
)
|
||||
pipeline.add_component("evaluator", evaluator)
|
||||
```
|
||||
|
||||
To run the evaluation pipeline, you should have the _expected inputs_ for the metric ready at hand. This metric expects a list of `questions` and `contexts`, which should come from the results of the pipeline you want to evaluate.
|
||||
|
||||
```python
|
||||
results = pipeline.run(
|
||||
{
|
||||
"evaluator": {
|
||||
"questions": [
|
||||
"When was the Rhodes Statue built?",
|
||||
"Where is the Pyramid of Giza?",
|
||||
],
|
||||
"contexts": [["Context for question 1"], ["Context for question 2"]],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
#### Evaluate Context Relevance and Aspect Critique
|
||||
|
||||
To create a pipeline that evaluates context relevance and aspect critique:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.evaluators.ragas import (
|
||||
RagasEvaluator,
|
||||
RagasMetric,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
evaluator_context = RagasEvaluator(
|
||||
metric=RagasMetric.CONTEXT_PRECISION,
|
||||
)
|
||||
evaluator_aspect = RagasEvaluator(
|
||||
metric=RagasMetric.ASPECT_CRITIQUE,
|
||||
metric_params={
|
||||
"name": "custom",
|
||||
"definition": "Is this answer problematic for children?",
|
||||
"strictness": 3,
|
||||
},
|
||||
)
|
||||
pipeline.add_component("evaluator_context", evaluator_context)
|
||||
pipeline.add_component("evaluator_aspect", evaluator_aspect)
|
||||
```
|
||||
|
||||
To run the evaluation pipeline, you should have the _expected inputs_ for the metrics ready at hand. These metrics expect a list of `questions`, `contexts`, `responses`, and `ground_truths`. These should come from the results of the pipeline you want to evaluate.
|
||||
|
||||
```python
|
||||
QUESTIONS = [
|
||||
"Which is the most popular global sport?",
|
||||
"Who created the Python language?",
|
||||
]
|
||||
CONTEXTS = [
|
||||
[
|
||||
"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people.",
|
||||
],
|
||||
[
|
||||
"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
|
||||
],
|
||||
]
|
||||
RESPONSES = [
|
||||
"Football is the most popular sport with around 4 billion followers worldwide",
|
||||
"Python language was created by Guido van Rossum.",
|
||||
]
|
||||
GROUND_TRUTHS = [
|
||||
"Football is the most popular sport",
|
||||
"Python language was created by Guido van Rossum.",
|
||||
]
|
||||
results = pipeline.run(
|
||||
{
|
||||
"evaluator_context": {
|
||||
"questions": QUESTIONS,
|
||||
"contexts": CONTEXTS,
|
||||
"ground_truths": GROUND_TRUTHS,
|
||||
},
|
||||
"evaluator_aspect": {
|
||||
"questions": QUESTIONS,
|
||||
"contexts": CONTEXTS,
|
||||
"responses": RESPONSES,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Evaluate a RAG pipeline using Ragas integration](https://haystack.deepset.ai/cookbook/rag_eval_ragas)
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "SASEvaluator"
|
||||
id: sasevaluator
|
||||
slug: "/sasevaluator"
|
||||
description: "The `SASEvaluator` evaluates answers predicted by Haystack pipelines using ground truth labels. It checks the semantic similarity of a predicted answer and the ground truth answer using a fine-tuned language model. This metric is called semantic answer similarity."
|
||||
---
|
||||
|
||||
# SASEvaluator
|
||||
|
||||
The `SASEvaluator` evaluates answers predicted by Haystack pipelines using ground truth labels. It checks the semantic similarity of a predicted answer and the ground truth answer using a fine-tuned language model. This metric is called semantic answer similarity.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | On its own or in an evaluation pipeline. To be used after a separate pipeline that has generated the inputs for the Evaluator. |
|
||||
| **Mandatory init variables** | `token`: A HF API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `ground_truth_answers`: A list of strings containing the ground truth answers <br /> <br />`predicted_answers`: A list of strings containing the predicted answers to be evaluated |
|
||||
| **Output variables** | A dictionary containing: <br /> <br />\- `score`: A number from 0.0 to 1.0 representing the mean SAS score for all pairs of predicted answers and ground truth answers <br /> <br />- `individual_scores`: A list of the SAS scores ranging from 0.0 to 1.0 of all pairs of predicted answers and ground truth answers |
|
||||
| **API reference** | [Evaluators](/reference/evaluators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/sas_evaluator.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
You can use the `SASEvaluator` component to evaluate answers predicted by a Haystack pipeline, such as a RAG pipeline, against ground truth labels.
|
||||
|
||||
You can provide a bi-encoder or cross-encoder model to initialize a `SASEvaluator`. By default, `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` model is used.
|
||||
|
||||
Note that only _one_ predicted answer is compared to _one_ ground truth answer at a time. The component does not support multiple ground truth answers for the same question or multiple answers predicted for the same question.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example of using a `SASEvaluator` component to evaluate two answers and compare them to ground truth answers.
|
||||
|
||||
```python
|
||||
from haystack.components.evaluators import SASEvaluator
|
||||
|
||||
sas_evaluator = SASEvaluator()
|
||||
result = sas_evaluator.run(
|
||||
ground_truth_answers=["Berlin", "Paris"],
|
||||
predicted_answers=["Berlin", "Lyon"],
|
||||
)
|
||||
print(result["individual_scores"])
|
||||
## [[array([[0.99999994]], dtype=float32), array([[0.51747656]], dtype=float32)]
|
||||
print(result["score"])
|
||||
## 0.7587383
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example where we use an `AnswerExactMatchEvaluator` and a `SASEvaluator` in a pipeline to evaluate two answers and compare them to ground truth answers. Running a pipeline instead of the individual components simplifies calculating more than one metric.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.evaluators import AnswerExactMatchEvaluator, SASEvaluator
|
||||
|
||||
pipeline = Pipeline()
|
||||
em_evaluator = AnswerExactMatchEvaluator()
|
||||
sas_evaluator = SASEvaluator()
|
||||
pipeline.add_component("em_evaluator", em_evaluator)
|
||||
pipeline.add_component("sas_evaluator", sas_evaluator)
|
||||
|
||||
ground_truth_answers = ["Berlin", "Paris"]
|
||||
predicted_answers = ["Berlin", "Lyon"]
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"em_evaluator": {
|
||||
"ground_truth_answers": ground_truth_answers,
|
||||
"predicted_answers": predicted_answers,
|
||||
},
|
||||
"sas_evaluator": {
|
||||
"ground_truth_answers": ground_truth_answers,
|
||||
"predicted_answers": predicted_answers,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["individual_scores"])
|
||||
## [1, 0]
|
||||
## [array([[0.99999994]], dtype=float32), array([[0.51747656]], dtype=float32)]
|
||||
|
||||
for evaluator in result:
|
||||
print(result[evaluator]["score"])
|
||||
## 0.5
|
||||
## 0.7587383
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Prompt Optimization with DSPy](https://haystack.deepset.ai/cookbook/prompt_optimization_with_dspy)
|
||||
Reference in New Issue
Block a user