chore: import upstream snapshot with attribution
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:25 +08:00
commit db620d33df
5151 changed files with 925932 additions and 0 deletions
@@ -0,0 +1,75 @@
# Foundry Evals Integration Samples
These samples demonstrate evaluating agent-framework agents using Azure AI Foundry's built-in evaluators.
## Available Evaluators
| Category | Evaluators |
|----------|-----------|
| **Agent behavior** | `intent_resolution`, `task_adherence`, `task_completion`, `task_navigation_efficiency` |
| **Tool usage** | `tool_call_accuracy`, `tool_selection`, `tool_input_accuracy`, `tool_output_utilization`, `tool_call_success` |
| **Quality** | `coherence`, `fluency`, `relevance`, `groundedness`, `response_completeness`, `similarity` |
| **Safety** | `violence`, `sexual`, `self_harm`, `hate_unfairness` |
## Samples
### `evaluate_agent_sample.py` — Dataset Evaluation (Path 3)
The dev inner loop. Two patterns from simplest to most control:
1. **`evaluate_agent()`** — One call: runs agent → converts → evaluates
2. **`FoundryEvals.evaluate()`** — Run agent yourself, convert with `AgentEvalConverter`, inspect/modify, then evaluate
```bash
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py
```
### `evaluate_traces_sample.py` — Trace & Response Evaluation (Path 1)
Evaluate what already happened — zero changes to agent code:
1. **`evaluate_traces(response_ids=...)`** — Evaluate Responses API responses by ID
2. **`evaluate_traces(agent_id=...)`** — Evaluate agent behavior from OTel traces in App Insights
```bash
uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py
```
### Referencing a rubric evaluator created in Foundry
Foundry users can create rubric evaluators in the Foundry portal (or
through the dedicated SDK / REST surface). Once an evaluator exists,
agent-framework consumes it like any other evaluator: pass a
`GeneratedEvaluatorRef(name=..., version=...)` in the `evaluators=`
list and pin the version for reproducible runs.
```python
from agent_framework.foundry import FoundryEvals, GeneratedEvaluatorRef
evals = FoundryEvals(
evaluators=[
GeneratedEvaluatorRef(name="reservation-policy-rubric", version="3"),
"relevance",
"coherence",
],
)
```
Quality gates on rubric output use the standard `EvalResults` helpers,
including `assert_dimension_score_at_least(...)` for per-dimension
thresholds.
See [`evaluate_with_rubric_sample.py`](./evaluate_with_rubric_sample.py)
for a runnable end-to-end example that combines a rubric evaluator with
built-in evaluators and gates a per-dimension threshold.
## Setup
Create a `.env` file with configuration as in the `.env.example` file in this folder.
## Which sample should I start with?
- **"I want to test my agent during development"** → `evaluate_agent_sample.py`, Pattern 1
- **"I want to evaluate past agent runs"** → `evaluate_traces_sample.py`
- **"I want to inspect/modify eval data before submitting"** → `evaluate_agent_sample.py`, Pattern 2
- **"I want to score against a custom rubric I created in Foundry"** → `evaluate_with_rubric_sample.py`