chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# Text-to-SQL Evaluation Quickstart
|
||||
|
||||
The `text2sql` template evaluates text-to-SQL systems by comparing SQL execution results.
|
||||
|
||||
## Create the Project
|
||||
|
||||
```sh
|
||||
ragas quickstart text2sql
|
||||
cd text2sql
|
||||
```
|
||||
|
||||
## Install Dependencies
|
||||
|
||||
```sh
|
||||
uv sync
|
||||
```
|
||||
|
||||
## Set Your API Key
|
||||
|
||||
```sh
|
||||
export OPENAI_API_KEY="your-openai-key"
|
||||
```
|
||||
|
||||
## Run the Evaluation
|
||||
|
||||
```sh
|
||||
uv run python evals.py
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
text2sql/
|
||||
├── README.md # Project documentation
|
||||
├── pyproject.toml # Project configuration
|
||||
├── text2sql_agent.py # Text-to-SQL agent
|
||||
├── db_utils.py # Database utilities
|
||||
├── evals.py # Evaluation workflow
|
||||
├── prompt.txt # Base prompt template
|
||||
├── prompt_v2.txt # Improved prompt v2
|
||||
├── prompt_v3.txt # Improved prompt v3
|
||||
├── __init__.py # Python package marker
|
||||
└── evals/
|
||||
├── datasets/
|
||||
│ └── booksql_sample.csv # Sample book database queries
|
||||
├── experiments/ # Evaluation results
|
||||
└── logs/ # Execution logs
|
||||
```
|
||||
|
||||
## What It Evaluates
|
||||
|
||||
The template evaluates text-to-SQL generation:
|
||||
|
||||
- **Agent**: Converts natural language to SQL queries
|
||||
- **Database**: Sample book database with authors, titles, genres
|
||||
- **Test Cases**: Natural language questions → expected SQL queries
|
||||
- **Metric**: Execution accuracy by comparing query results using datacompy
|
||||
|
||||
## Understanding the Code
|
||||
|
||||
### The Agent (`text2sql_agent.py`)
|
||||
|
||||
Converts natural language to SQL:
|
||||
|
||||
```python
|
||||
from text2sql_agent import Text2SQLAgent
|
||||
|
||||
agent = Text2SQLAgent(client=openai_client)
|
||||
sql = await agent.generate_sql("Find all books by Jane Austen")
|
||||
```
|
||||
|
||||
### The Evaluation (`evals.py`)
|
||||
|
||||
Compares execution results:
|
||||
|
||||
```python
|
||||
@discrete_metric(name="execution_accuracy", allowed_values=["correct", "incorrect"])
|
||||
def execution_accuracy(expected_sql: str, predicted_success: bool, predicted_result):
|
||||
# Executes both SQLs and compares results using datacompy
|
||||
# Returns "correct" if results match, "incorrect" otherwise
|
||||
```
|
||||
|
||||
## Test Data
|
||||
|
||||
The template includes `evals/datasets/booksql_sample.csv` with sample questions and expected SQL queries for a book database.
|
||||
|
||||
## Customization
|
||||
|
||||
### Use Your Own Database
|
||||
|
||||
Update `db_utils.py` to connect to your database:
|
||||
|
||||
```python
|
||||
def get_db_connection():
|
||||
return sqlite3.connect("your_database.db")
|
||||
```
|
||||
|
||||
### Try Different Prompts
|
||||
|
||||
The template includes three prompt versions in `prompt.txt`, `prompt_v2.txt`, and `prompt_v3.txt`. Test each to see which works best.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Agent Evaluation](agent_evals.md) - Evaluate AI agents
|
||||
- [Workflow Evaluation](workflow_eval.md) - Evaluate complex workflows
|
||||
Reference in New Issue
Block a user