chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,95 @@
|
||||
# Evaluate an AI agent
|
||||
|
||||
This tutorial demonstrates how to evaluate an AI agent using Ragas, specifically a mathematical agent that can solve complex expressions using atomic operations and function calling capabilities. By the end of this tutorial, you will learn how to evaluate and iterate on an agent using evaluation-driven development.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[User Input<br/>Math Expression] --> B[MathToolsAgent]
|
||||
|
||||
subgraph LLM Agent Loop
|
||||
B --> D{Need to use a Tool?}
|
||||
D -- Yes --> E[Call Tool<br/>add/sub/mul/div]
|
||||
E --> F[Tool Result]
|
||||
F --> B
|
||||
D -- No --> G[Emit Final Answer]
|
||||
end
|
||||
|
||||
G --> H[Final Answer]
|
||||
```
|
||||
|
||||
We will start by testing our simple agent that can solve mathematical expressions using atomic operations and function calling capabilities.
|
||||
|
||||
```bash
|
||||
python -m ragas_examples.agent_evals.agent
|
||||
```
|
||||
|
||||
Next, we will create a few sample expressions and expected outputs for our agent, then convert them to a CSV file.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
dataset = [
|
||||
{"expression": "(2 + 3) * (4 - 1)", "expected": 15},
|
||||
{"expression": "5 * (6 + 2)", "expected": 40},
|
||||
{"expression": "10 - (3 + 2)", "expected": 5},
|
||||
]
|
||||
|
||||
df = pd.DataFrame(dataset)
|
||||
df.to_csv("datasets/test_dataset.csv", index=False)
|
||||
```
|
||||
|
||||
To evaluate the performance of our agent, we will define a non-LLM metric that compares if our agent's output is within a certain tolerance of the expected output and returns 1/0 based on the comparison.
|
||||
|
||||
```python
|
||||
from ragas.metrics import numeric_metric
|
||||
from ragas.metrics.result import MetricResult
|
||||
|
||||
@numeric_metric(name="correctness")
|
||||
def correctness_metric(prediction: float, actual: float):
|
||||
"""Calculate correctness of the prediction."""
|
||||
if isinstance(prediction, str) and "ERROR" in prediction:
|
||||
return 0.0
|
||||
result = 1.0 if abs(prediction - actual) < 1e-5 else 0.0
|
||||
return MetricResult(value=result, reason=f"Prediction: {prediction}, Actual: {actual}")
|
||||
```
|
||||
|
||||
Next, we will write the experiment loop that will run our agent on the test dataset and evaluate it using the metric, and store the results in a CSV file.
|
||||
|
||||
```python
|
||||
from ragas import experiment
|
||||
|
||||
@experiment()
|
||||
async def run_experiment(row):
|
||||
expression = row["expression"]
|
||||
expected_result = row["expected"]
|
||||
|
||||
# Get the model's prediction
|
||||
prediction = math_agent.solve(expression)
|
||||
|
||||
# Calculate the correctness metric
|
||||
correctness = correctness_metric.score(prediction=prediction.get("result"), actual=expected_result)
|
||||
|
||||
return {
|
||||
"expression": expression,
|
||||
"expected_result": expected_result,
|
||||
"prediction": prediction.get("result"),
|
||||
"log_file": prediction.get("log_file"),
|
||||
"correctness": correctness.value
|
||||
}
|
||||
```
|
||||
|
||||
Now whenever you make a change to your agent, you can run the experiment and see how it affects the performance of your agent.
|
||||
|
||||
## Running the example end to end
|
||||
|
||||
1. Set up your OpenAI API key
|
||||
```bash
|
||||
export OPENAI_API_KEY="your_api_key_here"
|
||||
```
|
||||
|
||||
2. Run the evaluation
|
||||
```bash
|
||||
python -m ragas_examples.agent_evals.evals
|
||||
```
|
||||
|
||||
Voilà! You have successfully evaluated an AI agent using Ragas. You can now view the results by opening the `experiments/experiment_name.csv` file.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Tutorials
|
||||
|
||||
## Installing dependencies
|
||||
|
||||
1. Install ragas_examples
|
||||
|
||||
```bash
|
||||
pip install ragas[examples]
|
||||
```
|
||||
2. Setup your OpenAI API key
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY = "your_openai_api_key"
|
||||
```
|
||||
|
||||
## Tutorials
|
||||
|
||||
1. [Evaluate a prompt](prompt.md)
|
||||
2. [Evaluate a simple RAG system](rag.md)
|
||||
3. [Evaluate a AI Workflow](workflow.md)
|
||||
4. [Evaluate an AI Agent](agent.md)
|
||||
@@ -0,0 +1,125 @@
|
||||
# Prompt Evaluation
|
||||
|
||||
In this tutorial, we will write a simple evaluation pipeline to evaluate a prompt that is part of an AI system, here a movie review sentiment classifier. At the end of this tutorial you’ll learn how to evaluate and iterate on a single prompt using evaluation driven development.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["'This movie was amazing!<br/>Great acting and plot.'"] --> B["Classifier Prompt"]
|
||||
B --> C["Positive"]
|
||||
```
|
||||
|
||||
|
||||
We will start by testing a simple prompt that classifies movie reviews as positive or negative.
|
||||
|
||||
First, make sure you have installed ragas examples and setup your OpenAI API key:
|
||||
|
||||
```bash
|
||||
pip install ragas[examples]
|
||||
export OPENAI_API_KEY = "your_openai_api_key"
|
||||
```
|
||||
|
||||
Now test the prompt:
|
||||
|
||||
```bash
|
||||
python -m ragas_examples.prompt_evals.prompt
|
||||
```
|
||||
|
||||
This will test the input `"The movie was fantastic and I loved every moment of it!"` and should output `"positive"`.
|
||||
|
||||
> **💡 Quick Start**: If you want to see the complete evaluation in action, you can jump straight to the [end-to-end command](#running-the-example-end-to-end) that runs everything and generates the CSV results automatically.
|
||||
|
||||
Next, we will write down few sample inputs and expected outputs for our prompt. Then convert them to a CSV file.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
samples = [{"text": "I loved the movie! It was fantastic.", "label": "positive"},
|
||||
{"text": "The movie was terrible and boring.", "label": "negative"},
|
||||
{"text": "It was an average film, nothing special.", "label": "positive"},
|
||||
{"text": "Absolutely amazing! Best movie of the year.", "label": "positive"}]
|
||||
pd.DataFrame(samples).to_csv("datasets/test_dataset.csv", index=False)
|
||||
```
|
||||
|
||||
Now we need to have a way to measure the performance of our prompt in this task. We will define a metric that will compare the output of our prompt with the expected output and outputs pass/fail based on it.
|
||||
|
||||
```python
|
||||
from ragas.metrics import discrete_metric
|
||||
from ragas.metrics.result import MetricResult
|
||||
|
||||
@discrete_metric(name="accuracy", allowed_values=["pass", "fail"])
|
||||
def my_metric(prediction: str, actual: str):
|
||||
"""Calculate accuracy of the prediction."""
|
||||
return MetricResult(value="pass", reason="") if prediction == actual else MetricResult(value="fail", reason="")
|
||||
```
|
||||
|
||||
Next, we will write the experiment loop that will run our prompt on the test dataset and evaluate it using the metric, and store the results in a csv file.
|
||||
|
||||
```python
|
||||
from ragas import experiment
|
||||
|
||||
@experiment()
|
||||
async def run_experiment(row):
|
||||
|
||||
response = run_prompt(row["text"])
|
||||
score = my_metric.score(
|
||||
prediction=response,
|
||||
actual=row["label"]
|
||||
)
|
||||
|
||||
experiment_view = {
|
||||
**row,
|
||||
"response":response,
|
||||
"score":score.value,
|
||||
}
|
||||
return experiment_view
|
||||
```
|
||||
|
||||
Now whenever you make a change to your prompt, you can run the experiment and see how it affects the performance of your prompt.
|
||||
|
||||
### Passing Additional Parameters
|
||||
|
||||
You can pass additional parameters like models or configurations to your experiment function:
|
||||
|
||||
```python
|
||||
@experiment()
|
||||
async def run_experiment(row, model):
|
||||
response = run_prompt(row["text"], model=model)
|
||||
score = my_metric.score(
|
||||
prediction=response,
|
||||
actual=row["label"]
|
||||
)
|
||||
|
||||
experiment_view = {
|
||||
**row,
|
||||
"response": response,
|
||||
"score": score.value,
|
||||
}
|
||||
return experiment_view
|
||||
|
||||
# Run with specific parameters
|
||||
run_experiment.arun(dataset, "gpt-4")
|
||||
|
||||
# Or use keyword arguments
|
||||
run_experiment.arun(dataset, model="gpt-4o")
|
||||
```
|
||||
|
||||
|
||||
## Running the example end to end
|
||||
|
||||
1. Setup your OpenAI API key
|
||||
```bash
|
||||
export OPENAI_API_KEY = "your_openai_api_key"
|
||||
```
|
||||
2. Run the evaluation
|
||||
```bash
|
||||
python -m ragas_examples.prompt_evals.evals
|
||||
```
|
||||
|
||||
This will:
|
||||
|
||||
- Create the test dataset with sample movie reviews
|
||||
- Run the sentiment classification prompt on each sample
|
||||
- Evaluate the results using the accuracy metric
|
||||
- Export everything to a CSV file with the results
|
||||
|
||||
Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Evaluate a simple RAG system
|
||||
|
||||
In this tutorial, we will write a simple evaluation pipeline to evaluate a RAG (Retrieval-Augmented Generation) system. At the end of this tutorial, you’ll learn how to evaluate and iterate on a RAG system using evaluation-driven development.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Query<br/>'What is Ragas 0.3?'"] --> B[Retrieval System]
|
||||
|
||||
C[Document Corpus<br/> Ragas 0.3 Docs📄] --> B
|
||||
|
||||
B --> D[LLM + Prompt]
|
||||
A --> D
|
||||
|
||||
D --> E[Final Answer]
|
||||
```
|
||||
|
||||
We will start by writing a simple RAG system that retrieves relevant documents from a corpus and generates an answer using an LLM.
|
||||
|
||||
```bash
|
||||
python -m ragas_examples.rag_eval.rag
|
||||
```
|
||||
|
||||
|
||||
Next, we will write down a few sample queries and expected outputs for our RAG system. Then convert them to a CSV file.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
samples = [
|
||||
{"query": "What is Ragas 0.3?", "grading_notes": "- Ragas 0.3 is a library for evaluating LLM applications."},
|
||||
{"query": "How to install Ragas?", "grading_notes": "- install from source - install from pip using ragas[examples]"},
|
||||
{"query": "What are the main features of Ragas?", "grading_notes": "organised around - experiments - datasets - metrics."}
|
||||
]
|
||||
pd.DataFrame(samples).to_csv("datasets/test_dataset.csv", index=False)
|
||||
```
|
||||
|
||||
To evaluate the performance of our RAG system, we will define a llm based metric that compares the output of our RAG system with the grading notes and outputs pass/fail based on it.
|
||||
|
||||
```python
|
||||
from ragas.metrics import DiscreteMetric
|
||||
my_metric = DiscreteMetric(
|
||||
name="correctness",
|
||||
prompt = "Check if the response contains points mentioned from the grading notes and return 'pass' or 'fail'.\nResponse: {response} Grading Notes: {grading_notes}",
|
||||
allowed_values=["pass", "fail"],
|
||||
)
|
||||
```
|
||||
|
||||
Next, we will write the experiment loop that will run our RAG system on the test dataset and evaluate it using the metric, and store the results in a CSV file.
|
||||
|
||||
```python
|
||||
@experiment()
|
||||
async def run_experiment(row):
|
||||
response = rag_client.query(row["query"])
|
||||
|
||||
score = my_metric.score(
|
||||
llm=llm,
|
||||
response=response.get("answer", " "),
|
||||
grading_notes=row["grading_notes"]
|
||||
)
|
||||
|
||||
experiment_view = {
|
||||
**row,
|
||||
"response": response.get("answer", ""),
|
||||
"score": score.value,
|
||||
"log_file": response.get("logs", " "),
|
||||
}
|
||||
return experiment_view
|
||||
```
|
||||
|
||||
Now whenever you make a change to your RAG pipeline, you can run the experiment and see how it affects the performance of your RAG.
|
||||
|
||||
## Running the example end to end
|
||||
|
||||
1. Setup your OpenAI API key
|
||||
```bash
|
||||
export OPENAI_API_KEY="your_openai_api_key"
|
||||
```
|
||||
2. Run the evaluation
|
||||
```bash
|
||||
python -m ragas_examples.rag_eval.evals
|
||||
```
|
||||
|
||||
Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Evaluate an AI workflow
|
||||
|
||||
This tutorial demonstrates how to evaluate an AI workflow using Ragas, here a simple custom email support triage workflow. By the end of this tutorial, you will learn how to evaluate and iterate on a workflow using evaluation-driven development.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Email Query"] --> B["Rule based Info Extractor"]
|
||||
B --> C["Template + LLM Response"]
|
||||
C --> D["Email Reply"]
|
||||
```
|
||||
|
||||
We will start by testing our simple workflow that extracts the necessary information from an email, routes it to the correct template and generates response using an LLM.
|
||||
|
||||
```bash
|
||||
python -m ragas_examples.workflow_eval.workflow
|
||||
```
|
||||
|
||||
|
||||
Next, we will write down a few sample email queries and expected outputs for our workflow. Then convert them to a CSV file.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
dataset_dict = [
|
||||
{
|
||||
"email": "Hi, I'm getting error code XYZ-123 when using version 2.1.4 of your software. Please help!",
|
||||
"pass_criteria": "category Bug Report; product_version 2.1.4; error_code XYZ-123; response references both version and error code"
|
||||
},
|
||||
|
||||
{
|
||||
"email": "I need to dispute invoice #INV-2024-001 for 299.99 dollars. The charge seems incorrect.",
|
||||
"pass_criteria": "category Billing; invoice_number INV-2024-001; amount 299.99; response references invoice and dispute process"
|
||||
}]
|
||||
pd.DataFrame(dataset_dict).to_csv("datasets/test_dataset.csv", index=False)
|
||||
```
|
||||
|
||||
To evaluate the performance of our workflow, we will define a llm based metric that compares the output of our workflow with the pass criteria and outputs pass/fail based on it.
|
||||
|
||||
```python
|
||||
from ragas.metrics import DiscreteMetric
|
||||
|
||||
my_metric = DiscreteMetric(
|
||||
name="response_quality",
|
||||
prompt="Evaluate the response based on the pass criteria: {pass_criteria}. Does the response meet the criteria? Return 'pass' or 'fail'.\nResponse: {response}",
|
||||
allowed_values=["pass", "fail"],
|
||||
)
|
||||
```
|
||||
|
||||
Next, we will write the evaluation experiment loop that will run our workflow on the test dataset and evaluate it using the metric, and store the results in a CSV file.
|
||||
|
||||
```python
|
||||
from ragas import experiment
|
||||
|
||||
@experiment()
|
||||
async def run_experiment(row):
|
||||
response = workflow_client.process_email(
|
||||
row["email"]
|
||||
)
|
||||
|
||||
score = my_metric.score(
|
||||
llm=llm,
|
||||
response=response.get("response_template", " "),
|
||||
pass_criteria=row["pass_criteria"]
|
||||
)
|
||||
|
||||
experiment_view = {
|
||||
**row,
|
||||
"response": response.get("response_template", " "),
|
||||
"score": score.value,
|
||||
"score_reason": score.reason,
|
||||
}
|
||||
return experiment_view
|
||||
```
|
||||
|
||||
Now whenever you make a change to your workflow, you can run the experiment and see how it affects the performance of your workflow. Then compare it to the previous results to see how it has improved or degraded.
|
||||
|
||||
## Running the example end to end
|
||||
1. Setup your OpenAI API key
|
||||
```bash
|
||||
export OPENAI_API_KEY="your_openai_api_key"
|
||||
```
|
||||
|
||||
2. Run the experiment
|
||||
```bash
|
||||
python -m ragas_examples.workflow_eval.evals
|
||||
```
|
||||
|
||||
Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file.
|
||||
Reference in New Issue
Block a user