chore: import upstream snapshot with attribution
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# Foundry Evals Integration Samples
|
||||
|
||||
These samples demonstrate evaluating agent-framework agents using Azure AI Foundry's built-in evaluators.
|
||||
|
||||
## Available Evaluators
|
||||
|
||||
| Category | Evaluators |
|
||||
|----------|-----------|
|
||||
| **Agent behavior** | `intent_resolution`, `task_adherence`, `task_completion`, `task_navigation_efficiency` |
|
||||
| **Tool usage** | `tool_call_accuracy`, `tool_selection`, `tool_input_accuracy`, `tool_output_utilization`, `tool_call_success` |
|
||||
| **Quality** | `coherence`, `fluency`, `relevance`, `groundedness`, `response_completeness`, `similarity` |
|
||||
| **Safety** | `violence`, `sexual`, `self_harm`, `hate_unfairness` |
|
||||
|
||||
## Samples
|
||||
|
||||
### `evaluate_agent_sample.py` — Dataset Evaluation (Path 3)
|
||||
|
||||
The dev inner loop. Two patterns from simplest to most control:
|
||||
|
||||
1. **`evaluate_agent()`** — One call: runs agent → converts → evaluates
|
||||
2. **`FoundryEvals.evaluate()`** — Run agent yourself, convert with `AgentEvalConverter`, inspect/modify, then evaluate
|
||||
|
||||
```bash
|
||||
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py
|
||||
```
|
||||
|
||||
### `evaluate_traces_sample.py` — Trace & Response Evaluation (Path 1)
|
||||
|
||||
Evaluate what already happened — zero changes to agent code:
|
||||
|
||||
1. **`evaluate_traces(response_ids=...)`** — Evaluate Responses API responses by ID
|
||||
2. **`evaluate_traces(agent_id=...)`** — Evaluate agent behavior from OTel traces in App Insights
|
||||
|
||||
```bash
|
||||
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py
|
||||
```
|
||||
|
||||
### Referencing a rubric evaluator created in Foundry
|
||||
|
||||
Foundry users can create rubric evaluators in the Foundry portal (or
|
||||
through the dedicated SDK / REST surface). Once an evaluator exists,
|
||||
agent-framework consumes it like any other evaluator: pass a
|
||||
`GeneratedEvaluatorRef(name=..., version=...)` in the `evaluators=`
|
||||
list and pin the version for reproducible runs.
|
||||
|
||||
```python
|
||||
from agent_framework.foundry import FoundryEvals, GeneratedEvaluatorRef
|
||||
|
||||
evals = FoundryEvals(
|
||||
evaluators=[
|
||||
GeneratedEvaluatorRef(name="reservation-policy-rubric", version="3"),
|
||||
"relevance",
|
||||
"coherence",
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
Quality gates on rubric output use the standard `EvalResults` helpers,
|
||||
including `assert_dimension_score_at_least(...)` for per-dimension
|
||||
thresholds.
|
||||
|
||||
See [`evaluate_with_rubric_sample.py`](./evaluate_with_rubric_sample.py)
|
||||
for a runnable end-to-end example that combines a rubric evaluator with
|
||||
built-in evaluators and gates a per-dimension threshold.
|
||||
|
||||
## Setup
|
||||
|
||||
Create a `.env` file with configuration as in the `.env.example` file in this folder.
|
||||
|
||||
## Which sample should I start with?
|
||||
|
||||
- **"I want to test my agent during development"** → `evaluate_agent_sample.py`, Pattern 1
|
||||
- **"I want to evaluate past agent runs"** → `evaluate_traces_sample.py`
|
||||
- **"I want to inspect/modify eval data before submitting"** → `evaluate_agent_sample.py`, Pattern 2
|
||||
- **"I want to score against a custom rubric I created in Foundry"** → `evaluate_with_rubric_sample.py`
|
||||
Reference in New Issue
Block a user