chore: import upstream snapshot with attribution

python/samples/05-end-to-end/evaluation/foundry_evals/.env.example

@@ -0,0 +1,12 @@
+FOUNDRY_PROJECT_ENDPOINT="<your-project-endpoint>"
+FOUNDRY_MODEL="<your-model-deployment>"
+
+# Only needed for evaluate_with_rubric_sample.py — connects to the
+# pre-existing Foundry agent that the rubric evaluator was created against.
+FOUNDRY_AGENT_NAME="<your-agent-name>"
+FOUNDRY_AGENT_VERSION="<your-agent-version>"
+
+# Only needed for evaluate_with_rubric_sample.py — references a rubric
+# evaluator you created in Foundry. Pin the version for reproducible runs.
+FOUNDRY_RUBRIC_NAME="<your-rubric-name>"
+FOUNDRY_RUBRIC_VERSION="<your-rubric-version>"

python/samples/05-end-to-end/evaluation/foundry_evals/README.md

@@ -0,0 +1,75 @@
+# Foundry Evals Integration Samples
+
+These samples demonstrate evaluating agent-framework agents using Azure AI Foundry's built-in evaluators.
+
+## Available Evaluators
+
+| Category | Evaluators |
+|----------|-----------|
+| **Agent behavior** | `intent_resolution`, `task_adherence`, `task_completion`, `task_navigation_efficiency` |
+| **Tool usage** | `tool_call_accuracy`, `tool_selection`, `tool_input_accuracy`, `tool_output_utilization`, `tool_call_success` |
+| **Quality** | `coherence`, `fluency`, `relevance`, `groundedness`, `response_completeness`, `similarity` |
+| **Safety** | `violence`, `sexual`, `self_harm`, `hate_unfairness` |
+
+## Samples
+
+### `evaluate_agent_sample.py` — Dataset Evaluation (Path 3)
+
+The dev inner loop. Two patterns from simplest to most control:
+
+1. **`evaluate_agent()`** — One call: runs agent → converts → evaluates
+2. **`FoundryEvals.evaluate()`** — Run agent yourself, convert with `AgentEvalConverter`, inspect/modify, then evaluate
+
+```bash
+uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py
+```
+
+### `evaluate_traces_sample.py` — Trace & Response Evaluation (Path 1)
+
+Evaluate what already happened — zero changes to agent code:
+
+1. **`evaluate_traces(response_ids=...)`** — Evaluate Responses API responses by ID
+2. **`evaluate_traces(agent_id=...)`** — Evaluate agent behavior from OTel traces in App Insights
+
+```bash
+uv run samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py
+```
+
+### Referencing a rubric evaluator created in Foundry
+
+Foundry users can create rubric evaluators in the Foundry portal (or
+through the dedicated SDK / REST surface). Once an evaluator exists,
+agent-framework consumes it like any other evaluator: pass a
+`GeneratedEvaluatorRef(name=..., version=...)` in the `evaluators=`
+list and pin the version for reproducible runs.
+
+```python
+from agent_framework.foundry import FoundryEvals, GeneratedEvaluatorRef
+
+evals = FoundryEvals(
+    evaluators=[
+        GeneratedEvaluatorRef(name="reservation-policy-rubric", version="3"),
+        "relevance",
+        "coherence",
+    ],
+)
+```
+
+Quality gates on rubric output use the standard `EvalResults` helpers,
+including `assert_dimension_score_at_least(...)` for per-dimension
+thresholds.
+
+See [`evaluate_with_rubric_sample.py`](./evaluate_with_rubric_sample.py)
+for a runnable end-to-end example that combines a rubric evaluator with
+built-in evaluators and gates a per-dimension threshold.
+
+## Setup
+
+Create a `.env` file with configuration as in the `.env.example` file in this folder.
+
+## Which sample should I start with?
+
+- **"I want to test my agent during development"** → `evaluate_agent_sample.py`, Pattern 1
+- **"I want to evaluate past agent runs"** → `evaluate_traces_sample.py`
+- **"I want to inspect/modify eval data before submitting"** → `evaluate_agent_sample.py`, Pattern 2
+- **"I want to score against a custom rubric I created in Foundry"** → `evaluate_with_rubric_sample.py`

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_agent_sample.py

@@ -0,0 +1,190 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate an agent using Azure AI Foundry's built-in evaluators.
+
+This sample demonstrates three patterns:
+1. evaluate_agent(responses=...) — Evaluate a response you already have.
+2. evaluate_agent(queries=...) — Run the agent against test queries and evaluate in one call.
+3. Similarity — Compare agent output against ground-truth reference answers.
+
+See ``evaluate_tool_calls_sample.py`` for tool-call accuracy evaluation.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, ConversationSplit, evaluate_agent
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Define a simple tool for the agent
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    # 1. Set up the FoundryChatClient
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create an agent with tools
+    agent = Agent(
+        client=chat_client,
+        name="travel-assistant",
+        instructions=(
+            "You are a helpful travel assistant. Use your tools to answer questions about weather and flights."
+        ),
+        tools=[get_weather, get_flight_price],
+    )
+
+    # 3. Create the evaluator — provider config goes here, once
+    evals = FoundryEvals(client=chat_client)
+
+    # =========================================================================
+    # Pattern 1: evaluate_agent(responses=...) — evaluate a response you already have
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: evaluate_agent(responses=...) — evaluate existing response")
+    print("=" * 60)
+
+    query = "How much does a flight from Seattle to Paris cost?"
+    response = await agent.run(query)
+    print(f"Agent said: {response.text[:100]}...")
+
+    # Pass agent= so tool definitions are extracted, queries= for the eval item context
+    results = await evaluate_agent(
+        agent=agent,
+        responses=response,
+        queries=[query],
+        evaluators=FoundryEvals(
+            client=chat_client,
+            evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
+        ),
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 2a: evaluate_agent() — batch test queries
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2a: evaluate_agent()")
+    print("=" * 60)
+
+    # Calls agent.run() under the covers for each query, then evaluates
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "How much does a flight from Seattle to Paris cost?",
+            "What should I pack for London?",
+        ],
+        evaluators=evals,  # uses smart defaults (auto-adds tool_call_accuracy)
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 2b: evaluate_agent() — with conversation split override
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2b: evaluate_agent() with conversation_split")
+    print("=" * 60)
+
+    # conversation_split forces all evaluators to use the same split strategy.
+    # FULL evaluates the entire conversation trajectory against the original query.
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "What should I pack for London?",
+        ],
+        evaluators=evals,
+        conversation_split=ConversationSplit.FULL,  # overrides evaluator defaults
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 3: Similarity — compare agent output to ground-truth answers
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 3: Similarity evaluation with ground truth")
+    print("=" * 60)
+
+    # Similarity requires expected_output — a reference answer per query
+    # that the evaluator compares against the agent's actual response.
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "How much does a flight from Seattle to Paris cost?",
+        ],
+        expected_output=[
+            "62°F, cloudy with a chance of rain",
+            "Flights from Seattle to Paris: $450 round-trip",
+        ],
+        evaluators=FoundryEvals(
+            client=chat_client,
+            evaluators=[FoundryEvals.SIMILARITY],
+        ),
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_mixed_sample.py

@@ -0,0 +1,159 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Mix local and cloud evaluation providers in a single evaluate_agent() call.
+
+This sample demonstrates three patterns:
+1. Local-only: Fast, API-free checks for inner-loop development.
+2. Cloud-only: Full Foundry evaluators for comprehensive quality assessment.
+3. Mixed: Local + Foundry evaluators in a single evaluate_agent() call.
+
+Mixing lets you get instant local feedback (keyword presence, tool usage)
+alongside deeper cloud-based quality evaluation (relevance, coherence)
+in one call.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import (
+    Agent,
+    LocalEvaluator,
+    evaluate_agent,
+    keyword_check,
+    tool_called_check,
+)
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Define a simple tool for the agent
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create an agent with a tool
+    agent = Agent(
+        client=chat_client,
+        name="weather-assistant",
+        instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
+        tools=[get_weather],
+    )
+
+    # =========================================================================
+    # Pattern 1: Local evaluation only (no API calls, instant results)
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: Local evaluation only")
+    print("=" * 60)
+
+    local = LocalEvaluator(
+        keyword_check("weather", "seattle"),
+        tool_called_check("get_weather"),
+    )
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=["What's the weather in Seattle?"],
+        evaluators=local,
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        for check_name, counts in r.per_evaluator.items():
+            print(f"  {check_name}: {counts['passed']} passed, {counts['failed']} failed")
+        if r.all_passed:
+            print("[PASS] All local checks passed!")
+        else:
+            print(f"[FAIL] Failures: {r.error}")
+
+    # =========================================================================
+    # Pattern 2: Foundry evaluation only (cloud-based quality assessment)
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 2: Foundry evaluation only")
+    print("=" * 60)
+
+    foundry = FoundryEvals(client=chat_client)
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=["What's the weather in Seattle?"],
+        evaluators=foundry,
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Pattern 3: Mixed — local + Foundry in one call
+    # =========================================================================
+    print()
+    print("=" * 60)
+    print("Pattern 3: Mixed local + Foundry evaluation")
+    print("=" * 60)
+
+    # Local checks: fast smoke tests
+    local = LocalEvaluator(
+        keyword_check("weather"),
+        tool_called_check("get_weather"),
+    )
+
+    # Foundry: deep quality assessment
+    foundry = FoundryEvals(client=chat_client)
+
+    # Pass both as a list — returns one EvalResults per provider
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather in Seattle?",
+            "Tell me the weather in London",
+        ],
+        evaluators=[local, foundry],
+    )
+
+    for r in results:
+        status = "PASS" if r.all_passed else "FAIL"
+        print(f"  {status} {r.provider}: {r.passed}/{r.total} passed")
+        for check_name, counts in r.per_evaluator.items():
+            print(f"      {check_name}: {counts['passed']}/{counts['passed'] + counts['failed']}")
+        if r.report_url:
+            print(f"      Portal: {r.report_url}")
+
+    if all(r.all_passed for r in results):
+        print("[PASS] All checks passed (local + Foundry)!")
+    else:
+        failed = [r.provider for r in results if not r.all_passed]
+        print(f"[FAIL] Failed providers: {', '.join(failed)}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_multiturn_sample.py

@@ -0,0 +1,182 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate multi-turn conversations with different split strategies.
+
+The same multi-turn conversation can be split different ways, each evaluating
+a different aspect of agent behavior:
+
+1. LAST_TURN (default) — "Was the last response good given context?"
+2. FULL — "Did the whole conversation serve the original request?"
+3. per_turn_items — "Was each individual response appropriate?"
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Content, ConversationSplit, EvalItem, FunctionTool, Message
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# A multi-turn conversation with tool calls that we'll evaluate three ways.
+# Uses framework Message/Content types for type-safe conversation construction.
+CONVERSATION: list[Message] = [
+    # Turn 1: user asks about weather -> agent calls tool -> responds
+    Message("user", ["What's the weather in Seattle?"]),
+    Message(
+        "assistant",
+        [
+            Content.from_function_call("c1", "get_weather", arguments={"location": "seattle"}),
+        ],
+    ),
+    Message(
+        "tool",
+        [
+            Content.from_function_result("c1", result="62°F, cloudy with a chance of rain"),
+        ],
+    ),
+    Message("assistant", ["Seattle is 62°F, cloudy with a chance of rain."]),
+    # Turn 2: user asks about Paris -> agent calls tool -> responds
+    Message("user", ["And Paris?"]),
+    Message(
+        "assistant",
+        [
+            Content.from_function_call("c2", "get_weather", arguments={"location": "paris"}),
+        ],
+    ),
+    Message(
+        "tool",
+        [
+            Content.from_function_result("c2", result="68°F, partly sunny"),
+        ],
+    ),
+    Message("assistant", ["Paris is 68°F, partly sunny."]),
+    # Turn 3: user asks for comparison -> agent synthesizes without tool
+    Message("user", ["Can you compare them?"]),
+    Message(
+        "assistant",
+        [
+            (
+                "Seattle is cooler at 62°F with rain likely, while Paris is warmer "
+                "at 68°F and partly sunny. Paris is the better choice for outdoor activities."
+            ),
+        ],
+    ),
+]
+
+TOOLS = [
+    FunctionTool(
+        name="get_weather",
+        description="Get the current weather for a location.",
+    ),
+]
+
+
+def print_split(item: EvalItem, split: ConversationSplit = ConversationSplit.LAST_TURN) -> None:
+    """Print the query/response split for an EvalItem."""
+    query_msgs, response_msgs = item.split_messages(split)
+    print(f"  query_messages ({len(query_msgs)}):")
+    for m in query_msgs:
+        text = m.text or ""
+        print(f"    {m.role}: {text[:70]}")
+    print(f"  response_messages ({len(response_msgs)}):")
+    for m in response_msgs:
+        text = m.text or ""
+        print(f"    {m.role}: {text[:70]}")
+
+
+async def main() -> None:
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # =========================================================================
+    # Strategy 1: LAST_TURN (default)
+    # "Given all context, was the last response good?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 1: LAST_TURN — evaluate the final response")
+    print("=" * 70)
+
+    # EvalItem takes conversation + tools; query/response are derived via split strategy
+    item = EvalItem(CONVERSATION, tools=TOOLS)
+
+    print_split(item, ConversationSplit.LAST_TURN)
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        # conversation_split defaults to LAST_TURN
+    ).evaluate([item], eval_name="Split Strategy: LAST_TURN")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    # =========================================================================
+    # Strategy 2: FULL
+    # "Given the original request, did the whole conversation serve the user?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 2: FULL — evaluate the entire conversation trajectory")
+    print("=" * 70)
+
+    print_split(item, ConversationSplit.FULL)
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        conversation_split=ConversationSplit.FULL,
+    ).evaluate([item], eval_name="Split Strategy: FULL")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    # =========================================================================
+    # Strategy 3: per_turn_items
+    # "Was each individual response appropriate at that point?"
+    # =========================================================================
+    print("=" * 70)
+    print("Strategy 3: per_turn_items — evaluate each turn independently")
+    print("=" * 70)
+
+    items = EvalItem.per_turn_items(CONVERSATION, tools=TOOLS)
+    print(f"  Split into {len(items)} items from {len(CONVERSATION)} messages:\n")
+    for i, it in enumerate(items):
+        print(f"  Turn {i + 1}: query={it.query!r}, response={it.response[:60]!r}...")
+    print()
+
+    results = await FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+    ).evaluate(items, eval_name="Split Strategy: Per-Turn")
+
+    print(f"\n  Result: {results.passed}/{results.total} passed ({len(items)} items × 2 evaluators)")
+    print(f"  Portal: {results.report_url}")
+    for ir in results.items:
+        for s in ir.scores:
+            print(f"    {'PASS' if s.passed else 'FAIL'} {s.name}: {s.score}")
+    print()
+
+    print("=" * 70)
+    print("All strategies complete. Compare results in the Foundry portal.")
+    print("=" * 70)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_tool_calls_sample.py

@@ -0,0 +1,88 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate tool-calling accuracy using Azure AI Foundry's TOOL_CALL_ACCURACY evaluator.
+
+This sample demonstrates evaluating how well an agent selects and invokes tools
+by using ``FoundryEvals.evaluate()`` with ``TOOL_CALL_ACCURACY``.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, AgentEvalConverter
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # Create an agent with tools
+    agent = Agent(
+        client=chat_client,
+        name="travel-assistant",
+        instructions=(
+            "You are a helpful travel assistant. Use your tools to answer questions about weather and flights."
+        ),
+        tools=[get_weather, get_flight_price],
+    )
+
+    # Run the agent and convert responses to eval items
+    queries = [
+        "What's the weather in Paris?",
+        "Find me a flight from London to Seattle",
+    ]
+
+    items = []
+    for q in queries:
+        response = await agent.run(q)
+        print(f"Query: {q}")
+        print(f"Response: {response.text[:100]}...")
+
+        item = AgentEvalConverter.to_eval_item(query=q, response=response, agent=agent)
+        items.append(item)
+
+        print(f"  Has tools: {item.tools is not None}")
+        if item.tools:
+            print(f"  Tools: {[t.name for t in item.tools]}")
+
+    # Submit to Foundry with tool_call_accuracy evaluator
+    evals = FoundryEvals(
+        client=chat_client,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
+    )
+    results = await evals.evaluate(items, eval_name="Tool Call Accuracy Eval")
+
+    print(f"\nStatus: {results.status}")
+    print(f"Results: {results.passed}/{results.total} passed")
+    print(f"Portal: {results.report_url}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py

@@ -0,0 +1,114 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate agent responses that already exist in Foundry (zero-code-change).
+
+This sample demonstrates two patterns:
+1. evaluate_traces(response_ids=...) — Evaluate specific Responses API responses by ID.
+2. evaluate_traces(agent_id=...) — Evaluate agent behavior from OTel traces in App Insights.
+
+These are the "zero-code-change" evaluation paths — the agent has already run,
+and you're evaluating what happened after the fact.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Response IDs from prior agent runs (for Pattern 1)
+- OTel traces exported to App Insights (for Pattern 2)
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework.foundry import FoundryChatClient, FoundryEvals, evaluate_traces
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    chat_client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # =========================================================================
+    # Pattern 1: evaluate_traces(response_ids=...) — By response ID
+    # =========================================================================
+    # If your agent uses the Responses API (e.g., FoundryChatClient),
+    # each run produces a response_id. Pass those IDs to evaluate_traces()
+    # and Foundry retrieves the full conversation for evaluation.
+    print("=" * 60)
+    print("Pattern 1: evaluate_traces(response_ids=...)")
+    print("=" * 60)
+
+    # Replace these with actual response IDs from your agent runs
+    response_ids = [
+        "resp_abc123",
+        "resp_def456",
+    ]
+
+    results = await evaluate_traces(
+        response_ids=response_ids,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.GROUNDEDNESS, FoundryEvals.TOOL_CALL_ACCURACY],
+        client=chat_client,
+    )
+
+    print(f"Status: {results.status}")
+    print(f"Results: {results.result_counts}")
+    print(f"Portal: {results.report_url}")
+
+    # =========================================================================
+    # Pattern 2: evaluate_traces(response_ids=...) — Batch response evaluation
+    # =========================================================================
+    # Evaluate multiple prior responses by their IDs.  This uses the same
+    # response-based data source under the covers but lets you batch them.
+    #
+    # A future trace-based pattern (agent_id + lookback_hours) is shown
+    # commented out below — it requires OTel traces exported to App Insights.
+    print()
+    print("=" * 60)
+    print("Pattern 2: evaluate_traces(response_ids=...)")
+    print("=" * 60)
+
+    # Evaluate by response IDs (uses response-based data source internally)
+    results = await evaluate_traces(
+        response_ids=response_ids,
+        evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
+        client=chat_client,
+    )
+
+    print(f"Status: {results.status}")
+    print(f"Portal: {results.report_url}")
+
+    # Evaluate by agent ID + time window (when trace-based API is available)
+    # results = await evaluate_traces(
+    #     agent_id="travel-bot",
+    #     evaluators=[FoundryEvals.INTENT_RESOLUTION, FoundryEvals.TASK_ADHERENCE],
+    #     client=chat_client,
+    #     lookback_hours=24,
+    # )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+
+"""
+Sample output (with actual Azure AI Foundry project and valid response IDs):
+
+============================================================
+Pattern 1: evaluate_traces(response_ids=...)
+============================================================
+Status: completed
+Results: {'passed': 2, 'failed': 0, 'errored': 0}
+Portal: https://ai.azure.com/...
+
+============================================================
+Pattern 2: evaluate_traces(response_ids=...)
+============================================================
+Status: completed
+Portal: https://ai.azure.com/...
+"""

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_with_rubric_sample.py

@@ -0,0 +1,138 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate a Foundry agent against a rubric evaluator that was created in Foundry.
+
+Rubric evaluators are LLM-as-judge evaluators with custom scoring dimensions
+that you define for your domain. agent-framework consumes pre-existing rubric
+evaluators — they are authored in the Foundry portal (or via the dedicated
+SDK / REST surface) and referenced here by name and version.
+
+See: https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-evaluators/rubric-evaluators
+
+This sample demonstrates:
+1. Connecting to a pre-existing Foundry agent (PromptAgent or HostedAgent).
+2. Referencing a pre-existing rubric evaluator by ``name`` and ``version``.
+3. Mixing the rubric with built-in Foundry evaluators in one run.
+4. Asserting per-dimension thresholds with
+   ``EvalResults.assert_dimension_score_at_least(...)`` for CI quality gates.
+
+Starting condition / prerequisites:
+- An Azure AI Foundry project with a deployed model.
+- A registered Foundry agent (PromptAgent or HostedAgent) in that project.
+  This is the agent the rubric is meant to evaluate.
+- A rubric evaluator already created in the Foundry portal against that
+  agent. Creating rubrics through the portal currently requires picking a
+  Foundry agent as the generation context, so this prerequisite is implied
+  by having a rubric at all.
+- Set the following in .env (see ``.env.example``):
+    - ``FOUNDRY_PROJECT_ENDPOINT``
+    - ``FOUNDRY_AGENT_NAME`` and ``FOUNDRY_AGENT_VERSION`` for the agent
+    - ``FOUNDRY_RUBRIC_NAME`` and ``FOUNDRY_RUBRIC_VERSION`` for the rubric
+    - ``FOUNDRY_MODEL`` for the rubric judge model
+"""
+
+import asyncio
+import os
+
+from agent_framework import EvalNotPassedError, evaluate_agent
+from agent_framework.foundry import FoundryAgent, FoundryChatClient, FoundryEvals, GeneratedEvaluatorRef
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv(override=True)
+
+
+async def main() -> None:
+    # 1. Connect to the existing Foundry agent that the rubric was created
+    #    against. PromptAgents and HostedAgents are both supported.
+    credential = AzureCliCredential()
+    project_endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
+
+    agent = FoundryAgent(
+        project_endpoint=project_endpoint,
+        agent_name=os.environ["FOUNDRY_AGENT_NAME"],
+        agent_version=os.environ.get("FOUNDRY_AGENT_VERSION"),
+        credential=credential,
+    )
+
+    # 2. Reference the pre-existing rubric evaluator by name + version.
+    #    Always pin a version for reproducible CI runs; versionless refs
+    #    resolve to "latest" and emit a warning at evaluation time.
+    rubric_name = os.environ["FOUNDRY_RUBRIC_NAME"]
+    rubric_version = os.environ["FOUNDRY_RUBRIC_VERSION"]
+    rubric = GeneratedEvaluatorRef(name=rubric_name, version=rubric_version)
+
+    # 3. Mix the rubric with built-in evaluators in a single FoundryEvals
+    #    config. FoundryEvals talks to Foundry over the project endpoint, so
+    #    we hand it a FoundryChatClient configured with the same credential.
+    eval_client = FoundryChatClient(
+        project_endpoint=project_endpoint,
+        model=os.environ["FOUNDRY_MODEL"],
+        credential=credential,
+    )
+    evals = FoundryEvals(
+        client=eval_client,
+        evaluators=[
+            rubric,
+            FoundryEvals.RELEVANCE,
+            FoundryEvals.COHERENCE,
+        ],
+    )
+
+    # =========================================================================
+    # Run evaluation
+    # =========================================================================
+    print("=" * 60)
+    print(f"Evaluating '{agent.name}' with rubric '{rubric_name}' (version {rubric_version})")
+    print("=" * 60)
+
+    results = await evaluate_agent(
+        agent=agent,
+        queries=[
+            "What's the weather like in Seattle?",
+            "Should I bring an umbrella to London tomorrow?",
+        ],
+        evaluators=evals,
+    )
+
+    for r in results:
+        print(f"Status: {r.status}")
+        print(f"Results: {r.passed}/{r.total} passed")
+        print(f"Portal: {r.report_url}")
+        if r.all_passed:
+            print("[PASS] All passed")
+        else:
+            print(f"[FAIL] {r.failed} failed")
+
+    # =========================================================================
+    # Per-dimension quality gate
+    # =========================================================================
+    # Rubric evaluators emit per-dimension scores (1–5) on top of the overall
+    # weighted score. Use assert_dimension_score_at_least to gate CI on a
+    # specific dimension — e.g., never ship if a critical dimension drops
+    # below 3.
+    #
+    # The dimension_id must match an id defined on your rubric in Foundry.
+    # ``general_quality`` is used here because it's the conventional
+    # ``always_applicable: true`` dimension in the Foundry docs' example
+    # rubric — swap it for whatever dimension id(s) your rubric actually
+    # defines.
+    print()
+    print("=" * 60)
+    print("Per-dimension quality gate")
+    print("=" * 60)
+
+    for r in results:
+        try:
+            r.assert_dimension_score_at_least(
+                "general_quality",
+                min_score=3.0,
+                evaluator=rubric_name,
+            )
+            print(f"[PASS] {r.provider}: general_quality >= 3 on every item")
+        except EvalNotPassedError as exc:
+            print(f"[FAIL] {r.provider}: dimension gate tripped: {exc}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_workflow_sample.py

@@ -0,0 +1,221 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Evaluate a multi-agent workflow using Azure AI Foundry evaluators.
+
+This sample demonstrates three patterns:
+1. Post-hoc: Run the workflow, then evaluate the result you already have.
+2. Run + evaluate: Pass queries and let evaluate_workflow() run the workflow for you.
+3. Similarity: Evaluate the workflow's final output against ground-truth reference answers.
+
+Patterns 1 & 2 return a list of results (one per provider), each with a per-agent
+breakdown in sub_results so you can identify which agent is underperforming.
+
+Prerequisites:
+- An Azure AI Foundry project with a deployed model
+- Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env
+"""
+
+import asyncio
+import os
+
+from agent_framework import Agent, evaluate_workflow
+from agent_framework.foundry import FoundryChatClient, FoundryEvals
+from agent_framework_orchestrations import SequentialBuilder
+from azure.identity import AzureCliCredential
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+# Simple tools for the agents
+def get_weather(location: str) -> str:
+    """Get the current weather for a location."""
+    weather_data = {
+        "seattle": "62°F, cloudy with a chance of rain",
+        "london": "55°F, overcast",
+        "paris": "68°F, partly sunny",
+    }
+    return weather_data.get(location.lower(), f"Weather data not available for {location}")
+
+
+def get_flight_price(origin: str, destination: str) -> str:
+    """Get the price of a flight between two cities."""
+    return f"Flights from {origin} to {destination}: $450 round-trip"
+
+
+async def main() -> None:
+    # 1. Set up the chat client
+    client = FoundryChatClient(
+        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
+        credential=AzureCliCredential(),
+    )
+
+    # 2. Create agents for a sequential workflow
+    # Use store=False so agents don't chain conversation state via previous_response_id.
+    # This allows the workflow to be run multiple times without stale state issues.
+    researcher = Agent(
+        client=client,
+        name="researcher",
+        instructions=(
+            "You are a travel researcher. Use your tools to gather weather "
+            "and flight information for the destination the user asks about."
+        ),
+        tools=[get_weather, get_flight_price],
+        default_options={"store": False},
+    )
+
+    planner = Agent(
+        client=client,
+        name="planner",
+        instructions=(
+            "You are a travel planner. Based on the research provided, "
+            "create a concise travel recommendation with packing tips."
+        ),
+        default_options={"store": False},
+    )
+
+    # 3. Build a sequential workflow: researcher -> planner
+    workflow = SequentialBuilder(participants=[researcher, planner]).build()
+
+    # 4. Create the evaluator — provider config goes here, once
+    evals = FoundryEvals(client=client)
+    # =========================================================================
+    # Pattern 1: Post-hoc — evaluate a workflow run you already did
+    # =========================================================================
+    print("=" * 60)
+    print("Pattern 1: Post-hoc workflow evaluation")
+    print("=" * 60)
+
+    result = await workflow.run("Plan a trip from Seattle to Paris")
+
+    eval_results = await evaluate_workflow(
+        workflow=workflow,
+        workflow_result=result,
+        evaluators=evals,
+    )
+
+    for r in eval_results:
+        print(f"\nOverall: {r.status}")
+        print(f"  Passed: {r.passed}/{r.total}")
+        print(f"  Portal: {r.report_url}")
+
+        print("\nPer-agent breakdown:")
+        for agent_name, agent_eval in r.sub_results.items():
+            print(f"  {agent_name}: {agent_eval.passed}/{agent_eval.total} passed")
+            if agent_eval.report_url:
+                print(f"    Portal: {agent_eval.report_url}")
+
+    # =========================================================================
+    # Pattern 2: Run + evaluate with multiple queries
+    # =========================================================================
+    # Build a fresh workflow to avoid stale session state from Pattern 1.
+    # The Responses API tracks previous_response_id per session, so reusing
+    # a workflow after a run would reference stale tool calls.
+    workflow2 = SequentialBuilder(participants=[researcher, planner]).build()
+
+    print()
+    print("=" * 60)
+    print("Pattern 2: Run + evaluate with multiple queries")
+    print("=" * 60)
+
+    eval_results = await evaluate_workflow(
+        workflow=workflow2,
+        queries=[
+            "Plan a trip from London to Tokyo",
+            "Plan a trip from New York to Rome",
+        ],
+        evaluators=FoundryEvals(
+            client=client,
+            evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TASK_ADHERENCE],
+        ),
+    )
+
+    for r in eval_results:
+        print(f"\nOverall: {r.status}")
+        print(f"  Passed: {r.passed}/{r.total}")
+        if r.report_url:
+            print(f"  Portal: {r.report_url}")
+
+        print("\nPer-agent breakdown:")
+        for agent_name, agent_eval in r.sub_results.items():
+            print(f"  {agent_name}: {agent_eval.passed}/{agent_eval.total} passed")
+            if agent_eval.report_url:
+                print(f"    Portal: {agent_eval.report_url}")
+
+    # =========================================================================
+    # Pattern 3: Similarity — compare workflow output to ground-truth answers
+    # =========================================================================
+    # Build a fresh workflow to avoid stale session state from Pattern 2.
+    workflow3 = SequentialBuilder(participants=[researcher, planner]).build()
+
+    print()
+    print("=" * 60)
+    print("Pattern 3: Similarity evaluation with ground truth")
+    print("=" * 60)
+
+    # Similarity compares the final workflow output against a reference answer,
+    # so per-agent breakdown is disabled — individual agents don't have their
+    # own ground-truth targets.
+    eval_results = await evaluate_workflow(
+        workflow=workflow3,
+        queries=[
+            "Plan a trip from Seattle to Paris",
+            "Plan a trip from London to Tokyo",
+        ],
+        expected_output=[
+            "Pack layers and an umbrella for Paris. Flights from Seattle are around $450 round-trip.",
+            "Bring warm clothing for Tokyo in spring. Flights from London are around $500 round-trip.",
+        ],
+        evaluators=FoundryEvals(
+            client=client,
+            evaluators=[FoundryEvals.SIMILARITY],
+        ),
+        include_per_agent=False,
+    )
+
+    for r in eval_results:
+        print(f"\nOverall: {r.status}")
+        print(f"  Passed: {r.passed}/{r.total}")
+        if r.report_url:
+            print(f"  Portal: {r.report_url}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+
+
+"""
+Sample output (with actual Azure AI Foundry project):
+
+============================================================
+Pattern 1: Post-hoc workflow evaluation
+============================================================
+
+Overall: completed
+  Passed: 2/2
+  Portal: https://ai.azure.com/...
+
+Per-agent breakdown:
+  researcher: 1/1 passed
+  planner: 1/1 passed
+
+============================================================
+Pattern 2: Run + evaluate with multiple queries
+============================================================
+
+Overall: completed
+  Passed: 4/4
+
+Per-agent breakdown:
+  researcher: 2/2 passed
+  planner: 2/2 passed
+
+============================================================
+Pattern 3: Similarity evaluation with ground truth
+============================================================
+
+Overall: completed
+  Passed: 2/2
+  Portal: https://ai.azure.com/...
+"""