chore: import upstream snapshot with attribution

python/packages/lab/.gitignore

@@ -0,0 +1,13 @@
+test-results.xml
+
+# GAIA data directories
+data_gaia_hub/
+**/data_gaia_hub/
+gaia/**/*.jsonl
+
+# Lightning data directories
+lightning/**/data/tau2
+
+# TAU2 data directories
+tau2/**/data/
+tau2/**/results/

python/packages/lab/AGENTS.md

@@ -0,0 +1,22 @@
+# Lab Package (agent-framework-lab)
+
+Experimental packages for cutting-edge features including benchmarking, reinforcement learning, and research initiatives.
+
+## Structure
+
+This package contains experimental sub-packages:
+
+- `gaia/` - GAIA benchmark integration
+- `lightning/` - Lightning-based training utilities
+- `tau2/` - Tau-bench evaluation framework
+- `namespace/` - Experimental namespace utilities
+
+## Note
+
+Lab packages are experimental and may change frequently. They are not included in the standard `agent-framework[all]` installation.
+
+## Installation
+
+```bash
+pip install agent-framework-lab
+```

python/packages/lab/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

python/packages/lab/README.md

@@ -0,0 +1,119 @@
+# Agent Framework Lab
+
+This is the experimental package for Microsoft Agent Framework, `agent-framework-lab`, which contains
+various lab modules built on top of the core framework.
+Lab modules are not part of the core framework and may experience breaking changes or be deprecated in the future.
+
+## What are Lab Modules?
+
+Lab modules are extensions to the core Agent Framework that fall into
+one of the following categories:
+
+1. Incubation of new features that may get incorporated by the core framework.
+2. Research prototypes built on the core framework.
+3. Benchmarks and experimentation tools.
+
+## Lab Modules
+
+- [**gaia**](./gaia/): Evaluate your agents using the GAIA benchmark for general assistant tasks
+- [**tau2**](./tau2/): Evaluate your agents using the TAU2 benchmark for customer support tasks
+- [**lightning**](./lightning/): RL training for agents using Agent Lightning
+
+## Repository Structure
+
+```
+agent-framework-lab/
+├── pyproject.toml          # Single package configuration for agent-framework-lab
+├── README.md               # This file
+├── LICENSE                 # License file
+├── namespace/              # Centralized namespace package files
+│   └── agent_framework/
+│       └── lab/
+│           ├── gaia/       # Re-exports from agent_framework_lab_gaia
+│           ├── lightning/  # Re-exports from agent_framework_lab_lightning
+│           └── tau2/       # Re-exports from agent_framework_lab_tau2
+├── gaia/                   # GAIA module implementation
+│   └── agent_framework_lab_gaia/
+├── lightning/              # Lightning module implementation
+│   └── agent_framework_lab_lightning/
+└── tau2/                   # TAU2 module implementation
+    └── agent_framework_lab_tau2/
+```
+
+This structure maintains a single PyPI package `agent-framework-lab` while supporting modular imports through the namespace package mechanism.
+
+## Installation
+
+To install each lab module, use the extras syntax with `pip`:
+
+```bash
+pip install "agent-framework-lab[gaia]"
+pip install "agent-framework-lab[tau2]"
+pip install "agent-framework-lab[lightning]"
+```
+
+## Usage
+
+Import and use lab modules from the `agent_framework.lab` namespace.
+For example, to use the GAIA module:
+
+```python
+# Using GAIA module
+from agent_framework.lab.gaia import GAIA
+```
+
+## Running Tests Locally
+
+For machine-safe local runs, prefer package-scoped commands first:
+
+```bash
+uv run --directory packages/lab poe test
+uv run --directory packages/lab pytest -q -m "not integration"
+```
+
+When you need to run lab tests from the repository root, scope the root task to the lab package:
+
+```bash
+uv run poe test -P lab
+```
+
+Lightning observability tests intentionally exercise heavier tracing paths and are marked as `resource_intensive`:
+
+```bash
+uv run --directory packages/lab pytest lightning/tests/test_lightning.py -m "resource_intensive" -q
+```
+
+## Should I consume Lab Modules?
+
+If you are looking for stable and production-ready features, you should not use lab modules. Stick to the core framework.
+
+If you are looking for experimentation, research, or want to
+benchmark different approaches -- most importantly, if you don't mind breaking changes and potential deprecations --
+then lab modules are for you.
+
+## Contributing to Lab Modules
+
+### Microsoft-maintained modules
+
+For Microsoft-maintained modules in this repository, please follow standard contribution guidelines and submit pull requests directly to this repository.
+
+### Community modules
+
+If you want to contribute a community-maintained lab module:
+
+1. Create a new repository on GitHub for your module
+2. Tag your repository with `agent-framework-lab` for discoverability
+3. Submit a PR to add a link to your repository in the [Lab Modules](#lab-modules) section above
+4. Use the PR title format: `[New Lab Module] Your Module Name`
+
+We will review your submission based on the guidelines below.
+
+### Guidelines
+
+1. **Purpose**: Community modules should fit into one of the three categories of lab modules (incubation, research, benchmarks)
+2. **Namespace**: Community modules should avoid the `agent_framework.lab` namespace (reserved for modules maintained in this repository)
+3. **Dependencies**: Minimize external dependencies, always include `agent-framework` as a base dependency
+4. **Documentation**: Include comprehensive README with installation instructions and usage examples
+5. **Tests**: Write comprehensive tests with good coverage
+6. **Type hints**: Always include type hints and a `py.typed` file
+7. **Versioning**: Use semantic versioning, start with `0.1.0` for initial releases

python/packages/lab/gaia/README.md

@@ -0,0 +1,52 @@
+# Agent Framework Lab - GAIA
+
+The GAIA benchmark can be used for evaluating agents and workflows built using the Agent Framework.
+It includes built-in benchmarks as well as utilities for running custom evaluations.
+
+> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `gaia` extra to use this module.
+
+## Setup
+
+Install the `agent-framework-lab` package with GAIA dependencies:
+
+```bash
+pip install "agent-framework-lab[gaia]"
+```
+
+Set up Hugging Face token:
+
+```bash
+export HF_TOKEN="hf\*..." # must have access to gaia-benchmark/GAIA
+```
+
+## Create an evaluation script
+
+Create a Python script (e.g., `run_gaia.py`) with the following content:
+
+```python
+from agent_framework.lab.gaia import GAIA, Task, Prediction, GAIATelemetryConfig
+
+async def run_task(task: Task) -> Prediction:
+    return Prediction(prediction="answer here", messages=[])
+
+async def main() -> None:
+    # Optional: Enable telemetry for detailed tracing
+    telemetry_config = GAIATelemetryConfig(
+        enable_tracing=True,
+        trace_to_file=True,
+        file_path="gaia_traces.jsonl"
+    )
+
+    runner = GAIA(telemetry_config=telemetry_config)
+    await runner.run(run_task, level=1, max_n=5, parallel=2)
+```
+
+See the [gaia_sample.py](./samples/gaia_sample.py) for more detail.
+
+## View results
+
+We provide a console viewer for reading GAIA results:
+
+```bash
+uv run gaia_viewer "gaia_results_<timestamp>.jsonl" --detailed
+```

python/packages/lab/gaia/agent_framework_lab_gaia/__init__.py

@@ -0,0 +1,26 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""GAIA benchmark module for Agent Framework."""
+
+import importlib.metadata
+
+from ._types import Evaluation, Evaluator, Prediction, Task, TaskResult, TaskRunner
+from .gaia import GAIA, GAIATelemetryConfig, gaia_scorer, viewer_main
+
+try:
+    __version__ = importlib.metadata.version(__name__)
+except importlib.metadata.PackageNotFoundError:
+    __version__ = "0.0.0"  # Fallback for development mode
+
+__all__ = [
+    "GAIA",
+    "Evaluation",
+    "Evaluator",
+    "GAIATelemetryConfig",
+    "Prediction",
+    "Task",
+    "TaskResult",
+    "TaskRunner",
+    "gaia_scorer",
+    "viewer_main",
+]

python/packages/lab/gaia/agent_framework_lab_gaia/_types.py

@@ -0,0 +1,79 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Common types for agent evaluation."""
+
+from dataclasses import dataclass
+from typing import Any, Protocol, runtime_checkable
+
+__all__ = [
+    "Evaluation",
+    "Evaluator",
+    "Prediction",
+    "Task",
+    "TaskResult",
+    "TaskRunner",
+]
+
+
+@dataclass
+class Task:
+    """Represents a task to be evaluated."""
+
+    task_id: str
+    question: str
+    answer: str | None = None
+    level: int | None = None
+    file_name: str | None = None
+    metadata: dict[str, Any] | None = None
+
+
+@dataclass
+class Prediction:
+    """Represents a prediction made by an agent for a task."""
+
+    prediction: str
+    messages: list[Any] | None = None
+    metadata: dict[str, Any] | None = None
+
+    def __post_init__(self) -> None:
+        if self.messages is None:
+            self.messages = []
+
+
+@dataclass
+class Evaluation:
+    """Represents the evaluation result of a prediction."""
+
+    is_correct: bool
+    score: float
+    details: dict[str, Any] | None = None
+
+
+@dataclass
+class TaskResult:
+    """Complete result for a single task evaluation."""
+
+    task_id: str
+    task: Task
+    prediction: Prediction
+    evaluation: Evaluation
+    runtime_seconds: float | None = None
+    error: str | None = None
+
+
+@runtime_checkable
+class TaskRunner(Protocol):
+    """Protocol for running tasks."""
+
+    async def __call__(self, task: Task) -> Prediction:
+        """Run a single task and return the prediction."""
+        ...
+
+
+@runtime_checkable
+class Evaluator(Protocol):
+    """Protocol for evaluating predictions."""
+
+    async def __call__(self, task: Task, prediction: Prediction) -> Evaluation:
+        """Evaluate a prediction for a given task."""
+        ...

python/packages/lab/gaia/agent_framework_lab_gaia/gaia.py

@@ -0,0 +1,712 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""GAIA benchmark implementation for Agent Framework."""
+
+import asyncio
+import json
+import os
+import random
+import re
+import string
+import tempfile
+import time
+from collections.abc import Callable, Iterable
+from datetime import datetime
+from functools import lru_cache
+from pathlib import Path
+from typing import Any, Protocol, cast
+
+from opentelemetry.trace import NoOpTracer, SpanKind, get_tracer
+from tqdm import tqdm
+
+from ._types import Evaluation, Evaluator, Prediction, Task, TaskResult, TaskRunner
+
+__all__ = ["GAIA", "GAIATelemetryConfig", "gaia_scorer"]
+
+
+class _OrjsonModule(Protocol):
+    def dumps(self, obj: object, /, default: Callable[[Any], object] | None = None) -> bytes: ...
+
+    def loads(self, obj: str | bytes | bytearray, /) -> object: ...
+
+
+@lru_cache(maxsize=1)
+def _get_orjson() -> _OrjsonModule | None:
+    try:
+        import orjson as runtime_orjson
+    except ImportError:
+        return None
+    return cast(_OrjsonModule, runtime_orjson)
+
+
+def _dump_json_line(value: object) -> str:
+    if (runtime_orjson := _get_orjson()) is not None:
+        return runtime_orjson.dumps(value, default=str).decode("utf-8")
+    return json.dumps(value, default=str)
+
+
+def _load_json_value(value: str | bytes) -> object:
+    if (runtime_orjson := _get_orjson()) is not None:
+        return runtime_orjson.loads(value)
+    return json.loads(value)
+
+
+class GAIATelemetryConfig:
+    """Configuration for GAIA telemetry and tracing."""
+
+    def __init__(
+        self,
+        enable_tracing: bool = False,
+        otlp_endpoint: str | None = None,
+        trace_to_file: bool = False,
+        file_path: str | None = None,
+    ):
+        """Initialize telemetry configuration.
+
+        Args:
+            enable_tracing: Whether to enable OpenTelemetry tracing
+            otlp_endpoint: OTLP endpoint for trace export
+            trace_to_file: Whether to export traces to local file
+            file_path: Path for local file export (defaults to gaia_traces.json)
+
+        Note:
+            For Azure Monitor integration, configure using environment variables
+            (OTEL_EXPORTER_OTLP_ENDPOINT, etc.) or call ``configure_azure_monitor()``
+            before creating the GAIA instance.
+        """
+        self.enable_tracing = enable_tracing
+        self.otlp_endpoint = otlp_endpoint
+        self.trace_to_file = trace_to_file
+        self.file_path = file_path or "gaia_traces.json"
+
+    def configure_otel_providers(self) -> None:
+        """Set up OpenTelemetry based on configuration."""
+        if not self.enable_tracing:
+            return
+
+        # If only file tracing is requested (no OTLP),
+        # skip the default configure_otel_providers which adds console exporter
+        if self.trace_to_file and not self.otlp_endpoint:
+            # Set up minimal tracing with only file export
+            from opentelemetry.sdk.trace import TracerProvider
+            from opentelemetry.trace import set_tracer_provider
+
+            tracer_provider = TracerProvider()
+            set_tracer_provider(tracer_provider)
+            self._setup_file_export()
+        else:
+            # Use full observability setup for OTLP
+            from agent_framework.observability import configure_otel_providers
+
+            # Set OTLP endpoint env var if provided
+            if self.otlp_endpoint:
+                import os
+
+                os.environ.setdefault("OTEL_EXPORTER_OTLP_ENDPOINT", self.otlp_endpoint)
+
+            configure_otel_providers(
+                enable_sensitive_data=True,  # Enable for detailed task traces
+            )
+
+            # Set up local file export if requested
+            if self.trace_to_file:
+                self._setup_file_export()
+
+    def _setup_file_export(self) -> None:
+        """Set up local file export for traces."""
+        try:
+            import json
+            import os
+            from collections.abc import Sequence
+
+            from opentelemetry.sdk.trace import ReadableSpan, TracerProvider
+            from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult
+            from opentelemetry.trace import get_tracer_provider
+
+            class FileSpanExporter(SpanExporter):
+                def __init__(self, file_path: str):
+                    self.file_path = file_path
+                    # Ensure directory exists
+                    os.makedirs(os.path.dirname(os.path.abspath(file_path)), exist_ok=True)
+
+                def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
+                    try:
+                        with open(self.file_path, "a", encoding="utf-8") as f:
+                            for span in spans:
+                                span_data = {
+                                    "trace_id": format(span.context.trace_id, "032x") if span.context else "unknown",
+                                    "span_id": format(span.context.span_id, "016x") if span.context else "unknown",
+                                    "name": span.name,
+                                    "start_time": span.start_time,
+                                    "end_time": span.end_time,
+                                    "duration_ns": (span.end_time - span.start_time)
+                                    if (span.end_time and span.start_time)
+                                    else None,
+                                    "attributes": dict(span.attributes) if span.attributes else {},
+                                    "status": {
+                                        "status_code": span.status.status_code.name if span.status else "UNSET",
+                                        "description": span.status.description if span.status else None,
+                                    },
+                                }
+                                f.write(json.dumps(span_data, default=str) + "\n")
+                        return SpanExportResult.SUCCESS
+                    except Exception:
+                        return SpanExportResult.FAILURE
+
+                def shutdown(self) -> None:
+                    pass
+
+            tracer_provider = get_tracer_provider()
+            if isinstance(tracer_provider, TracerProvider):
+                file_exporter = FileSpanExporter(self.file_path)
+                tracer_provider.add_span_processor(BatchSpanProcessor(file_exporter))
+
+        except ImportError:
+            print("Warning: Could not set up file export for traces. Missing dependencies.")
+
+
+def _normalize_number_str(number_str: str) -> float:
+    """Normalize a number string for comparison."""
+    for ch in ["$", "%", ","]:
+        number_str = number_str.replace(ch, "")
+    try:
+        return float(number_str)
+    except ValueError:
+        return float("inf")
+
+
+def _split_string(s: str, chars: list[str] | None = None) -> list[str]:
+    """Split string by multiple delimiters."""
+    if chars is None:
+        chars = [",", ";"]
+    return re.split(f"[{''.join(chars)}]", s)
+
+
+def _normalize_str(s: str, remove_punct: bool = True) -> str:
+    """Normalize string for comparison."""
+    no_spaces = re.sub(r"\s", "", s or "")
+    if remove_punct:
+        table = str.maketrans("", "", string.punctuation)
+        return no_spaces.lower().translate(table)
+    return no_spaces.lower()
+
+
+def gaia_scorer(model_answer: str | None, ground_truth: str) -> bool:
+    """Official GAIA scoring function.
+
+    Args:
+        model_answer: The model's answer
+        ground_truth: The ground truth answer
+
+    Returns:
+        True if the answer is correct, False otherwise
+    """
+
+    def is_float(x: Any) -> bool:
+        try:
+            float(x)
+            return True
+        except Exception:
+            return False
+
+    if model_answer is None:
+        model_answer = "None"
+
+    if is_float(ground_truth):
+        # numeric exact match after normalization
+        return abs(_normalize_number_str(model_answer) - float(ground_truth)) < 1e-6
+    if any(ch in ground_truth for ch in [",", ";"]):
+        # list with per-element compare (number or string)
+        gt_elems = _split_string(ground_truth)
+        ma_elems = _split_string(model_answer)
+        if len(gt_elems) != len(ma_elems):
+            return False
+        comparisons: list[bool] = []
+        for ma, gt in zip(ma_elems, gt_elems, strict=False):
+            if is_float(gt):
+                comparisons.append(abs(_normalize_number_str(ma) - float(gt)) < 1e-6)
+            else:
+                comparisons.append(_normalize_str(ma, remove_punct=False) == _normalize_str(gt, remove_punct=False))
+        return all(comparisons)
+    # string normalize + exact
+    return _normalize_str(model_answer) == _normalize_str(ground_truth)
+
+
+def _coerce_record(raw: object) -> dict[str, Any] | None:
+    if isinstance(raw, dict):
+        raw_dict = cast(dict[object, Any], raw)
+        if all(isinstance(key, str) for key in raw_dict):
+            return cast(dict[str, Any], raw_dict)
+    return None
+
+
+def _parse_level(level: object) -> int | None:
+    if isinstance(level, int):
+        return level
+    if isinstance(level, str) and level.isdigit():
+        return int(level)
+    return None
+
+
+def _read_jsonl(path: Path) -> Iterable[dict[str, Any]]:
+    """Read JSONL file and yield parsed records."""
+    with path.open("rb") as f:
+        for line in f:
+            if not line.strip():
+                continue
+            parsed = _load_json_value(line)
+
+            record = _coerce_record(parsed)
+            if record is not None:
+                yield record
+
+
+def _load_gaia_local(repo_dir: Path, wanted_levels: list[int] | None = None, max_n: int | None = None) -> list[Task]:
+    """Load GAIA tasks from local repository directory."""
+    tasks: list[Task] = []
+
+    # First try to load from parquet files (new format)
+    # Prioritize validation split over test split (validation has answers)
+    parquet_files = sorted(
+        repo_dir.rglob("metadata*.parquet"), key=lambda p: (0 if "validation" in str(p) else 1, str(p))
+    )
+
+    for p in parquet_files:
+        try:
+            import pyarrow.parquet as pq
+
+            pq_any = cast(Any, pq)
+            table: Any = pq_any.read_table(p)
+            rows = cast(list[object], table.to_pylist())
+            for row in rows:
+                record = _coerce_record(row)
+                if record is None:
+                    continue
+
+                # Robustly extract fields used across variants
+                q_obj = record.get("Question") or record.get("question") or record.get("query") or record.get("prompt")
+                ans = record.get("Final answer") or record.get("answer") or record.get("final_answer")
+                if not isinstance(q_obj, str):
+                    continue
+                q = q_obj
+
+                qid = str(
+                    record.get("task_id")
+                    or record.get("question_id")
+                    or record.get("id")
+                    or record.get("uuid")
+                    or f"{p.stem}:{len(tasks)}"
+                )
+                lvl = _parse_level(record.get("Level") or record.get("level"))
+                fname_obj = record.get("file_name") or record.get("filename")
+                fname = fname_obj if isinstance(fname_obj, str) else None
+
+                # Only evaluate examples with public answers (dev/validation split)
+                # Skip if no question, no answer, or answer is placeholder like "?"
+                if ans is None or str(ans).strip() in ["?", ""]:
+                    continue
+
+                if wanted_levels and (lvl not in wanted_levels):
+                    continue
+
+                tasks.append(
+                    Task(task_id=qid, question=q, answer=str(ans), level=lvl, file_name=fname, metadata=record)
+                )
+        except ImportError:
+            print("Warning: pyarrow not installed. Install with: pip install pyarrow")
+            continue
+        except Exception as e:
+            print(f"Warning: Could not load parquet file {p}: {e}")
+            continue
+
+    # Fall back to jsonl files (old format) if no parquet files found
+    if not tasks:
+        for p in repo_dir.rglob("metadata.jsonl"):
+            for rec in _read_jsonl(p):
+                # Robustly extract fields used across variants
+                q_obj = rec.get("Question") or rec.get("question") or rec.get("query") or rec.get("prompt")
+                ans = rec.get("Final answer") or rec.get("answer") or rec.get("final_answer")
+                if not isinstance(q_obj, str):
+                    continue
+                q = q_obj
+
+                qid = str(
+                    rec.get("task_id")
+                    or rec.get("question_id")
+                    or rec.get("id")
+                    or rec.get("uuid")
+                    or f"{p.stem}:{len(tasks)}"
+                )
+                lvl = _parse_level(rec.get("Level") or rec.get("level"))
+                fname_obj = rec.get("file_name") or rec.get("filename")
+                fname = fname_obj if isinstance(fname_obj, str) else None
+
+                # Only evaluate examples with public answers (dev/validation split)
+                # Skip if no question, no answer, or answer is placeholder like "?"
+                if ans is None or str(ans).strip() in ["?", ""]:
+                    continue
+
+                if wanted_levels and (lvl not in wanted_levels):
+                    continue
+
+                tasks.append(Task(task_id=qid, question=q, answer=str(ans), level=lvl, file_name=fname, metadata=rec))
+
+    # Shuffle to help with rate-limits and fairness if max_n is provided
+    random.shuffle(tasks)
+    if max_n:
+        tasks = tasks[:max_n]
+    return tasks
+
+
+class GAIA:
+    """GAIA benchmark runner for Agent Framework.
+
+    GAIA (General AI Assistant) is a benchmark for general-purpose AI assistants.
+    This class provides utilities to run the benchmark with custom agents.
+    """
+
+    def __init__(
+        self,
+        evaluator: Evaluator | None = None,
+        data_dir: str | None = None,
+        hf_token: str | None = None,
+        telemetry_config: GAIATelemetryConfig | None = None,
+    ):
+        """Initialize GAIA benchmark runner.
+
+        Args:
+            evaluator: Custom evaluator function. If None, uses default GAIA scorer.
+            data_dir: Directory to cache GAIA data. Defaults to a temporary directory.
+            hf_token: Hugging Face token for accessing the GAIA dataset.
+            telemetry_config: Configuration for telemetry and tracing. If None, no tracing is performed.
+        """
+        self.evaluator = evaluator or self._default_evaluator
+        self.data_dir = Path(data_dir or Path(tempfile.gettempdir()) / "data_gaia_hub")
+        self.hf_token = hf_token
+        self.telemetry_config = telemetry_config or GAIATelemetryConfig()
+
+        # Set up telemetry
+        self.telemetry_config.configure_otel_providers()
+
+        # Initialize tracer
+        if self.telemetry_config.enable_tracing:
+            self.tracer = get_tracer("gaia_benchmark", "1.0.0")
+        else:
+            self.tracer = NoOpTracer()
+
+    async def _default_evaluator(self, task: Task, prediction: Prediction) -> Evaluation:
+        """Default evaluator using GAIA official scoring."""
+        is_correct = gaia_scorer(prediction.prediction, task.answer or "")
+        return Evaluation(is_correct=is_correct, score=1.0 if is_correct else 0.0)
+
+    def _ensure_data(self) -> Path:
+        """Ensure GAIA data is available locally."""
+        if self.data_dir.exists() and any(self.data_dir.rglob("metadata.jsonl")):
+            return self.data_dir
+
+        # Download data if not available
+        token = self.hf_token or os.environ.get("HF_TOKEN")
+        if not token:
+            raise RuntimeError(
+                "HF_TOKEN environment variable or hf_token parameter is required "
+                "to access the GAIA dataset. Please set your Hugging Face token "
+                "with access to gaia-benchmark/GAIA."
+            )
+
+        import huggingface_hub
+
+        hf_hub = cast(Any, huggingface_hub)
+        local_dir = hf_hub.snapshot_download(
+            repo_id="gaia-benchmark/GAIA",
+            repo_type="dataset",
+            revision="682dd723ee1e1697e00360edccf2366dc8418dd9",
+            token=token,
+            local_dir=str(self.data_dir),
+            force_download=False,
+        )
+        if not isinstance(local_dir, str):
+            raise TypeError("snapshot_download returned unexpected non-string path")
+        return Path(local_dir)
+
+    async def _run_single_task(
+        self, task: Task, task_runner: TaskRunner, semaphore: asyncio.Semaphore, timeout: int | None = None
+    ) -> TaskResult:
+        """Run a single task with error handling and timing."""
+        async with semaphore:
+            with self.tracer.start_as_current_span(
+                "gaia.task.run",
+                kind=SpanKind.INTERNAL,
+                attributes={
+                    "gaia.task.id": task.task_id,
+                    "gaia.task.level": task.level or 0,
+                    "gaia.task.has_file": task.file_name is not None,
+                    "gaia.task.timeout": timeout or 0,
+                },
+            ) as span:
+                start_time = time.time()
+                try:
+                    # Add task execution span
+                    with self.tracer.start_as_current_span(
+                        "gaia.task.execute",
+                        kind=SpanKind.INTERNAL,
+                        attributes={
+                            "gaia.task.question_length": len(task.question or ""),
+                            "gaia.task.file_name": task.file_name or "",
+                        },
+                    ):
+                        if timeout:
+                            prediction = await asyncio.wait_for(task_runner(task), timeout=timeout)
+                        else:
+                            prediction = await task_runner(task)
+
+                    # Add evaluation span
+                    with self.tracer.start_as_current_span("gaia.task.evaluate", kind=SpanKind.INTERNAL):
+                        evaluation = await self.evaluator(task, prediction)
+
+                    runtime_seconds = time.time() - start_time
+
+                    # Add results to span
+                    if span:
+                        span.set_attributes({
+                            "gaia.task.runtime_seconds": runtime_seconds,
+                            "gaia.task.is_correct": evaluation.is_correct,
+                            "gaia.task.score": evaluation.score,
+                            "gaia.task.prediction_length": len(prediction.prediction or ""),
+                        })
+
+                    return TaskResult(
+                        task_id=task.task_id,
+                        task=task,
+                        prediction=prediction,
+                        evaluation=evaluation,
+                        runtime_seconds=runtime_seconds,
+                    )
+                except Exception as e:
+                    runtime_seconds = time.time() - start_time
+
+                    # Record error in span
+                    if span:
+                        span.set_attributes({
+                            "gaia.task.runtime_seconds": runtime_seconds,
+                            "gaia.task.error": str(e),
+                            "gaia.task.is_correct": False,
+                            "gaia.task.score": 0.0,
+                        })
+                        span.record_exception(e)
+
+                    return TaskResult(
+                        task_id=task.task_id,
+                        task=task,
+                        prediction=Prediction(prediction="", messages=[]),
+                        evaluation=Evaluation(is_correct=False, score=0.0),
+                        runtime_seconds=runtime_seconds,
+                        error=str(e),
+                    )
+
+    async def run(
+        self,
+        task_runner: TaskRunner,
+        level: int | list[int] = 1,
+        max_n: int | None = None,
+        parallel: int = 1,
+        timeout: int | None = None,
+        out: str | None = None,
+    ) -> list[TaskResult]:
+        """Run the GAIA benchmark.
+
+        Args:
+            task_runner: Function that takes a Task and returns a Prediction
+            level: GAIA level(s) to run (1, 2, 3, or list of levels)
+            max_n: Maximum number of tasks to run per level
+            parallel: Number of parallel tasks to run
+            timeout: Timeout per task in seconds
+            out: Output file to save results including detailed traces (optional)
+
+        Returns:
+            List of TaskResult objects
+        """
+        with self.tracer.start_as_current_span(
+            "gaia.benchmark.run",
+            kind=SpanKind.INTERNAL,
+            attributes={
+                "gaia.benchmark.levels": str(level),
+                "gaia.benchmark.max_n": max_n or 0,
+                "gaia.benchmark.parallel": parallel,
+                "gaia.benchmark.timeout": timeout or 0,
+            },
+        ) as benchmark_span:
+            # Ensure data is available
+            with self.tracer.start_as_current_span("gaia.data.ensure", kind=SpanKind.INTERNAL):
+                data_path = self._ensure_data()
+
+            # Parse level parameter
+            levels = [level] if isinstance(level, int) else level
+
+            # Load tasks
+            with self.tracer.start_as_current_span(
+                "gaia.tasks.load",
+                kind=SpanKind.INTERNAL,
+                attributes={
+                    "gaia.tasks.levels": str(levels),
+                    "gaia.tasks.max_n": max_n or 0,
+                },
+            ) as load_span:
+                tasks = _load_gaia_local(data_path, wanted_levels=levels, max_n=max_n)
+
+                if load_span:
+                    load_span.set_attributes({
+                        "gaia.tasks.loaded_count": len(tasks),
+                    })
+
+            if not tasks:
+                raise RuntimeError(
+                    f"No GAIA tasks found for levels {levels}. "
+                    "Make sure you have dataset access and selected valid levels."
+                )
+
+            # Update benchmark span with task info
+            if benchmark_span:
+                benchmark_span.set_attributes({
+                    "gaia.benchmark.total_tasks": len(tasks),
+                })
+
+            # Run tasks
+            semaphore = asyncio.Semaphore(parallel)
+            results: list[TaskResult] = []
+
+            tasks_coroutines = [self._run_single_task(task, task_runner, semaphore, timeout) for task in tasks]
+
+            with self.tracer.start_as_current_span("gaia.tasks.execute_all", kind=SpanKind.INTERNAL):
+                for coro in tqdm(
+                    asyncio.as_completed(tasks_coroutines), total=len(tasks_coroutines), desc="Evaluating tasks"
+                ):
+                    result = await coro
+                    results.append(result)
+
+            # Calculate summary statistics
+            correct = sum(1 for r in results if r.evaluation.is_correct)
+            accuracy = correct / len(results) if results else 0.0
+            avg_runtime = sum(r.runtime_seconds or 0 for r in results) / len(results) if results else 0.0
+
+            # Update benchmark span with final results
+            if benchmark_span:
+                benchmark_span.set_attributes({
+                    "gaia.benchmark.accuracy": accuracy,
+                    "gaia.benchmark.correct_count": correct,
+                    "gaia.benchmark.total_count": len(results),
+                    "gaia.benchmark.avg_runtime_seconds": avg_runtime,
+                })
+
+            # Save results if requested
+            if out:
+                with self.tracer.start_as_current_span(
+                    "gaia.results.save", kind=SpanKind.INTERNAL, attributes={"gaia.results.output_file": out}
+                ):
+                    self._save_results(results, out)
+
+            return results
+
+    def _save_results(self, results: list[TaskResult], output_path: str) -> None:
+        """Save results with detailed trace information to JSONL file."""
+        with open(output_path, "w", encoding="utf-8") as f:
+            for result in results:
+                # Convert messages to serializable format
+                serializable_messages: list[dict[str, Any] | str] = []
+                if result.prediction.messages:
+                    for msg in result.prediction.messages:
+                        if hasattr(msg, "model_dump"):
+                            # Pydantic model
+                            serializable_messages.append(msg.model_dump())
+                        elif hasattr(msg, "__dict__"):
+                            # Regular object with attributes
+                            serializable_messages.append(cast(dict[str, Any], getattr(msg, "__dict__", {})))
+                        else:
+                            # Fallback to string representation
+                            serializable_messages.append(str(msg))
+
+                record = {
+                    "task_id": result.task_id,
+                    "level": result.task.level,
+                    "question": result.task.question,
+                    "answer": result.task.answer,
+                    "prediction": result.prediction.prediction,
+                    "is_correct": result.evaluation.is_correct,
+                    "score": result.evaluation.score,
+                    "runtime_seconds": result.runtime_seconds,
+                    "error": result.error,
+                    "timestamp": datetime.now().isoformat(),
+                    # Include detailed trace information
+                    "task_metadata": result.task.metadata,
+                    "file_name": result.task.file_name,
+                    "messages": serializable_messages,
+                    "prediction_metadata": result.prediction.metadata,
+                    "evaluation_details": result.evaluation.details,
+                }
+                f.write(_dump_json_line(record) + "\n")
+
+
+def viewer_main() -> None:
+    """Main function for the gaia_viewer script."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="View GAIA benchmark results")
+    parser.add_argument("results_file", help="Path to results JSONL file")
+    parser.add_argument("--detailed", action="store_true", help="Show detailed view")
+    parser.add_argument("--level", type=int, help="Filter by level")
+    parser.add_argument("--correct-only", action="store_true", help="Show only correct answers")
+    parser.add_argument("--incorrect-only", action="store_true", help="Show only incorrect answers")
+
+    args = parser.parse_args()
+
+    # Load results
+    results: list[dict[str, Any]] = []
+    with open(args.results_file, encoding="utf-8") as f:
+        for line in f:
+            if line.strip():
+                parsed = _load_json_value(line)
+                record = _coerce_record(parsed)
+                if record is not None:
+                    results.append(record)
+
+    # Apply filters
+    if args.level is not None:
+        results = [r for r in results if r.get("level") == args.level]
+
+    if args.correct_only:
+        results = [r for r in results if r.get("is_correct")]
+    elif args.incorrect_only:
+        results = [r for r in results if not r.get("is_correct")]
+
+    # Display results
+    if not results:
+        print("No results match the filters.")
+        return
+
+    total = len(results)
+    correct = sum(1 for r in results if r.get("is_correct"))
+    accuracy = correct / total if total > 0 else 0.0
+
+    print("GAIA Results Summary:")
+    print(f"Total: {total}, Correct: {correct}, Accuracy: {accuracy:.3f}")
+    print("-" * 80)
+
+    for i, result in enumerate(results, 1):
+        status = "✓" if result.get("is_correct") else "✗"
+        level = result.get("level", "?")
+        task_id = result.get("task_id", "unknown")
+
+        print(f"[{i}/{total}] {status} Level {level} - {task_id}")
+
+        if args.detailed:
+            print(f"Question: {result.get('question', 'N/A')[:100]}...")
+            print(f"Answer: {result.get('answer', 'N/A')}")
+            print(f"Prediction: {result.get('prediction', 'N/A')}")
+            if result.get("error"):
+                print(f"Error: {result.get('error')}")
+            if result.get("runtime_seconds"):
+                print(f"Runtime: {result.get('runtime_seconds'):.2f}s")
+            print("-" * 40)
+
+
+if __name__ == "__main__":
+    viewer_main()

python/packages/lab/gaia/agent_framework_lab_gaia/py.typed

@@ -0,0 +1 @@
+py.typed

python/packages/lab/gaia/samples/azure_ai_agent.py

@@ -0,0 +1,67 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Azure AI Agent factory for GAIA benchmark.
+
+This module provides a factory function to create an Azure AI agent
+configured for GAIA benchmark tasks.
+
+Required Environment Variables:
+    FOUNDRY_PROJECT_ENDPOINT: Azure AI project endpoint URL
+    FOUNDRY_MODEL: Name of the model deployment to use
+
+Optional Environment Variables:
+    BING_CONNECTION_ID: ID of the Bing connection for web search
+
+Authentication:
+    Uses Azure CLI credentials via AzureCliCredential.
+    Run `az login` before executing to authenticate.
+
+Example:
+    export FOUNDRY_PROJECT_ENDPOINT="https://your-project.azure.com"
+    export FOUNDRY_MODEL="gpt-4o"
+    export BING_CONNECTION_ID="connection-id"
+    az login
+"""
+
+import os
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+
+from agent_framework import Agent
+from agent_framework.foundry import FoundryChatClient
+from azure.identity.aio import AzureCliCredential
+
+
+@asynccontextmanager
+async def create_gaia_agent() -> AsyncIterator[Agent]:
+    """Create an Azure AI agent configured for GAIA benchmark tasks.
+
+    The agent is configured with:
+    - Bing Search tool for web information retrieval
+    - Code Interpreter tool for calculations and data analysis
+
+    Yields:
+        Agent: A configured agent ready to run GAIA tasks.
+
+    Example:
+        async with create_gaia_agent() as agent:
+            result = await agent.run("What is the capital of France?")
+            print(result.text)
+    """
+    async with (
+        AzureCliCredential() as credential,
+        FoundryChatClient(
+            project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
+            model=os.environ["FOUNDRY_MODEL"],
+            credential=credential,
+        ).as_agent(
+            name="GaiaAgent",
+            instructions="Solve tasks to your best ability. Use Bing Search to find "
+            "information and Code Interpreter to perform calculations and data analysis.",
+            tools=[
+                FoundryChatClient.get_web_search_tool(),
+                FoundryChatClient.get_code_interpreter_tool(),
+            ],
+        ) as agent,
+    ):
+        yield agent

python/packages/lab/gaia/samples/gaia_sample.py

@@ -0,0 +1,295 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""GAIA Benchmark Sample.
+
+Run the GAIA (General AI Assistant) benchmark with configurable agent providers,
+telemetry options, and benchmark parameters.
+
+Agent Providers:
+    - Azure AI (default): See azure_ai_agent.py for required environment variables
+    - OpenAI: See openai_agent.py for required environment variables
+
+Prerequisites:
+    1. Set HF_TOKEN environment variable with your Hugging Face token:
+       - Get token: https://huggingface.co/settings/tokens
+       - Request dataset access: https://huggingface.co/datasets/gaia-benchmark/GAIA
+       - Set: export HF_TOKEN="your-huggingface-token"
+
+    2. Configure your chosen agent provider (see agent module files for details)
+
+Telemetry:
+    When using --otlp-endpoint or --trace-file, OpenTelemetry will export trace data
+    in JSON format to the console in addition to the configured endpoints. This is
+    expected behavior from the OpenTelemetry SDK and provides visibility into the
+    telemetry being captured. The traces are also exported to:
+    - OTLP endpoint (e.g., Aspire Dashboard) if --otlp-endpoint is specified
+    - Local file if --trace-file is specified
+
+    To suppress console output, redirect stderr: `python gaia_sample.py 2>/dev/null`
+
+Usage:
+    # Run with default settings (Azure AI agent)
+    uv run python gaia_sample.py
+
+    # Run with OpenAI agent
+    uv run python gaia_sample.py --agent-provider openai
+
+    # Run with telemetry export to Aspire Dashboard
+    uv run python gaia_sample.py --otlp-endpoint http://localhost:4318
+
+    # See all options
+    uv run python gaia_sample.py --help
+"""
+
+import argparse
+
+from agent_framework.lab.gaia import GAIA, Evaluation, GAIATelemetryConfig, Prediction, Task
+
+
+async def evaluate_task(task: Task, prediction: Prediction) -> Evaluation:
+    """Evaluate the prediction for a given task."""
+    # Simple evaluation: check if the prediction contains the answer
+    is_correct = (task.answer or "").lower() in prediction.prediction.lower()
+    return Evaluation(is_correct=is_correct, score=1 if is_correct else 0)
+
+
+async def main(
+    otlp_endpoint: str | None = None,
+    trace_file: str | None = None,
+    result_file: str | None = None,
+    data_dir: str | None = None,
+    agent_provider: str = "azure-ai",
+    level: int | list[int] = 1,
+    max_n: int = 2,
+    parallel: int = 1,
+    timeout: int = 120,
+) -> None:
+    """Run GAIA benchmark with telemetry configuration.
+
+    Args:
+        otlp_endpoint: Optional OTLP endpoint URL for exporting traces (e.g., http://localhost:4318)
+        trace_file: Optional file path to export traces to. If None, traces won't be saved to file.
+        result_file: Optional file path to save benchmark results. If None, results won't be saved to file.
+        data_dir: Directory to cache GAIA dataset. If None, uses temp directory.
+        agent_provider: Agent provider to use: 'azure-ai' or 'openai' (default: 'azure-ai')
+        level: GAIA level(s) to run (1, 2, or 3)
+        max_n: Maximum number of tasks to run per level
+        parallel: Number of parallel tasks to run
+        timeout: Timeout per task in seconds
+    """
+    # Check for required Hugging Face token
+    import logging
+    import os
+
+    # Suppress console logging for traces and verbose SDK output
+    logging.getLogger("opentelemetry").setLevel(logging.ERROR)
+    logging.getLogger("azure").setLevel(logging.WARNING)
+    logging.getLogger("agent_framework").setLevel(logging.WARNING)
+    logging.getLogger("httpx").setLevel(logging.WARNING)
+    logging.getLogger("httpcore").setLevel(logging.WARNING)
+
+    # Suppress OpenTelemetry exporters console output
+    import os as _os
+
+    _os.environ.setdefault("OTEL_PYTHON_LOG_LEVEL", "error")
+
+    # Print trace export configuration
+    print("\n=== Telemetry Configuration ===")
+    if trace_file:
+        print(f"📁 Trace file: {os.path.abspath(trace_file)}")
+    else:
+        print("📁 Trace file: disabled")
+
+    if otlp_endpoint:
+        print(f"🌐 OTLP endpoint: {otlp_endpoint}")
+    else:
+        print("🌐 OTLP endpoint: disabled")
+
+    if result_file:
+        print(f"📊 Results file: {os.path.abspath(result_file)}")
+    else:
+        print("📊 Results file: disabled")
+
+    print("\n=== Run Configuration ===")
+    print(f"🤖 Agent provider: {agent_provider}")
+    if data_dir:
+        print(f"📂 Data directory: {os.path.abspath(data_dir)}")
+    else:
+        import tempfile
+        from pathlib import Path
+
+        default_data_dir = Path(tempfile.gettempdir()) / "data_gaia_hub"
+        print(f"📂 Data directory: {default_data_dir} (default)")
+    print(f"🎯 Level: {level}")
+    print(f"🔢 Max tasks: {max_n}")
+    print(f"⚡ Parallel: {parallel}")
+    print(f"⏱️  Timeout: {timeout}s")
+    print()
+
+    # Import the appropriate agent factory based on provider
+    if agent_provider == "azure-ai":
+        from azure_ai_agent import create_gaia_agent
+    elif agent_provider == "openai":
+        from openai_agent import create_gaia_agent
+    else:
+        raise ValueError(f"Unknown agent provider: {agent_provider}. Use 'azure-ai' or 'openai'.")
+
+    # Configure telemetry for tracing
+    telemetry_config = GAIATelemetryConfig(
+        enable_tracing=True,  # Enable OpenTelemetry tracing
+        trace_to_file=trace_file is not None,  # Export traces to local file only if path provided
+        file_path=trace_file,  # Custom file path for traces (can be None)
+        otlp_endpoint=otlp_endpoint,  # Optional OTLP endpoint for Aspire Dashboard or other collectors
+    )
+
+    # Create a single agent once and reuse it for all tasks
+    async with create_gaia_agent() as agent:
+
+        async def run_task(task: Task) -> Prediction:
+            """Run a single GAIA task and return the prediction using the shared agent."""
+            input_message = f"Task: {task.question}"
+            if task.file_name:
+                input_message += f"\nFile: {task.file_name}"
+            result = await agent.run(input_message)
+            return Prediction(prediction=result.text, messages=result.messages)
+
+        # Create the GAIA benchmark runner with telemetry configuration
+        runner = GAIA(
+            evaluator=evaluate_task,
+            telemetry_config=telemetry_config,
+            data_dir=data_dir,
+        )
+
+        # Run the benchmark with the task runner.
+        # By default, this will check for locally cached benchmark data and checkout
+        # the latest version from HuggingFace if not found.
+        # Note: The GAIA dataset has been updated to use Parquet format.
+        # If you encounter issues, try using validation split which has labeled data.
+        results = await runner.run(
+            run_task,
+            level=level,
+            max_n=max_n,
+            parallel=parallel,
+            timeout=timeout,
+            out=result_file,  # Output file to save results including detailed traces (optional, None = no file output)
+        )
+
+    # Print summary similar to the viewer in gaia.py
+    total = len(results)
+    correct = sum(1 for r in results if r.evaluation.is_correct)
+    accuracy = correct / total if total > 0 else 0.0
+    avg_runtime = sum(r.runtime_seconds or 0 for r in results) / total if total > 0 else 0.0
+
+    print("\n=== GAIA Benchmark Summary ===")
+    print(f"📝 Total: {total}, ✅ Correct: {correct}, 🎯 Accuracy: {accuracy:.3f}")
+    print(f"⏱️  Average runtime: {avg_runtime:.2f}s")
+    if result_file:
+        print(f"💾 Detailed results saved to: {result_file}")
+
+
+if __name__ == "__main__":
+    import asyncio
+
+    # Parse command line arguments
+    parser = argparse.ArgumentParser(
+        description="Run GAIA benchmark with optional telemetry export to OTLP endpoint and/or file",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Run with default settings
+  python gaia_sample.py
+
+  # Run with custom data directory
+  python gaia_sample.py --data-dir ./gaia_data
+
+  # Run with OpenAI agent provider
+  python gaia_sample.py --agent-provider openai
+
+  # Run with trace file export
+  python gaia_sample.py --trace-file gaia_benchmark_traces.jsonl
+
+  # Run level 2 tasks with 5 maximum tasks
+  python gaia_sample.py --level 2 --max-n 5
+
+  # Run with OTLP export to Aspire Dashboard and custom settings
+  python gaia_sample.py --otlp-endpoint http://localhost:4318 --level 1 --max-n 10 --parallel 2
+
+  # Run with all options configured
+  python gaia_sample.py --agent-provider openai \
+  --trace-file traces.jsonl \
+  --result-file results.jsonl \
+  --otlp-endpoint http://localhost:4318 --level 1 --max-n 5 --parallel 2 --timeout 180
+        """,
+    )
+    parser.add_argument(
+        "--otlp-endpoint",
+        type=str,
+        default=None,
+        help="OTLP endpoint URL for exporting traces (e.g., http://localhost:4318 for Aspire Dashboard)",
+    )
+    parser.add_argument(
+        "--trace-file",
+        type=str,
+        default=None,
+        help="File path to export traces to (e.g., gaia_benchmark_traces.jsonl). "
+        "If not set, traces won't be saved to file.",
+    )
+    parser.add_argument(
+        "--result-file",
+        type=str,
+        default="gaia_results_level1.jsonl",
+        help="File path to save benchmark results (default: gaia_results_level1.jsonl)",
+    )
+    parser.add_argument(
+        "--data-dir",
+        type=str,
+        default=None,
+        help="Directory to cache GAIA dataset. If not set, uses system temp directory.",
+    )
+    parser.add_argument(
+        "--agent-provider",
+        type=str,
+        default="azure-ai",
+        choices=["azure-ai", "openai"],
+        help="Agent provider to use: 'azure-ai' or 'openai' (default: 'azure-ai')",
+    )
+    parser.add_argument(
+        "--level",
+        type=int,
+        default=1,
+        choices=[1, 2, 3],
+        help="GAIA benchmark level to run: 1, 2, or 3 (default: 1)",
+    )
+    parser.add_argument(
+        "--max-n",
+        type=int,
+        default=2,
+        help="Maximum number of tasks to run per level (default: 2)",
+    )
+    parser.add_argument(
+        "--parallel",
+        type=int,
+        default=1,
+        help="Number of parallel tasks to run (default: 1)",
+    )
+    parser.add_argument(
+        "--timeout",
+        type=int,
+        default=120,
+        help="Timeout per task in seconds (default: 120)",
+    )
+    args = parser.parse_args()
+
+    asyncio.run(
+        main(
+            otlp_endpoint=args.otlp_endpoint,
+            trace_file=args.trace_file,
+            result_file=args.result_file,
+            data_dir=args.data_dir,
+            agent_provider=args.agent_provider,
+            level=args.level,
+            max_n=args.max_n,
+            parallel=args.parallel,
+            timeout=args.timeout,
+        )
+    )

python/packages/lab/gaia/samples/openai_agent.py

@@ -0,0 +1,61 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""OpenAI Agent factory for GAIA benchmark.
+
+This module provides a factory function to create an OpenAI agent
+configured for GAIA benchmark tasks using the OpenAI Responses API.
+
+Required Environment Variables:
+    OPENAI_API_KEY: Your OpenAI API key
+    OPENAI_CHAT_MODEL: Model to use with Responses API (e.g., gpt-4o, gpt-4o-mini)
+
+Optional Environment Variables:
+    OPENAI_BASE_URL: Custom API base URL if using a proxy or compatible service
+    OPENAI_ORG_ID: Organization ID for OpenAI API (if applicable)
+
+Authentication:
+    Uses OPENAI_API_KEY environment variable.
+    Get your API key from: https://platform.openai.com/api-keys
+
+Example:
+    export OPENAI_API_KEY="sk-..."
+    export OPENAI_CHAT_MODEL="gpt-4o"
+"""
+
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+
+from agent_framework import Agent
+from agent_framework.openai import OpenAIChatClient
+
+
+@asynccontextmanager
+async def create_gaia_agent() -> AsyncIterator[Agent]:
+    """Create an OpenAI agent configured for GAIA benchmark tasks.
+
+    Uses OpenAI Responses API for enhanced capabilities.
+
+    The agent is configured with:
+    - Web Search tool for information retrieval
+    - Code Interpreter tool for calculations and data analysis
+
+    Yields:
+        Agent: A configured agent ready to run GAIA tasks.
+
+    Example:
+        async with create_gaia_agent() as agent:
+            result = await agent.run("What is the capital of France?")
+            print(result.text)
+    """
+    client = OpenAIChatClient()
+
+    async with client.as_agent(
+        name="GaiaAgent",
+        instructions="Solve tasks to your best ability. Use Web Search to find "
+        "information and Code Interpreter to perform calculations and data analysis.",
+        tools=[
+            OpenAIChatClient.get_web_search_tool(),
+            OpenAIChatClient.get_code_interpreter_tool(),
+        ],
+    ) as agent:
+        yield agent

python/packages/lab/gaia/tests/test_gaia.py

@@ -0,0 +1,36 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for GAIA benchmark implementation."""
+
+from agent_framework_lab_gaia import gaia_scorer
+
+
+class TestGAIAScorer:
+    """Test the GAIA scoring function."""
+
+    def test_numeric_exact_match(self):
+        """Test numeric exact matching."""
+        assert gaia_scorer("42", "42") is True
+        assert gaia_scorer("42.0", "42") is True
+        assert gaia_scorer("42", "42.0") is True
+        assert gaia_scorer("42", "43") is False
+
+    def test_string_normalization(self):
+        """Test string normalization and matching."""
+        assert gaia_scorer("Hello World", "hello world") is True
+        assert gaia_scorer("Hello, World!", "helloworld") is True
+        assert gaia_scorer("test", "TEST") is True
+        assert gaia_scorer("test", "different") is False
+
+    def test_list_matching(self):
+        """Test list matching with comma/semicolon separation."""
+        assert gaia_scorer("1,2,3", "1,2,3") is True
+        assert gaia_scorer("1; 2; 3", "1,2,3") is True
+        assert gaia_scorer("apple,banana", "apple,banana") is True
+        assert gaia_scorer("1,2,3", "1,2,4") is False
+        assert gaia_scorer("1,2", "1,2,3") is False
+
+    def test_none_handling(self):
+        """Test handling of None values."""
+        assert gaia_scorer("None", "test") is False
+        assert gaia_scorer("", "test") is False

python/packages/lab/lightning/.gitattributes

@@ -0,0 +1,2 @@
+assets/ filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

python/packages/lab/lightning/README.md

@@ -0,0 +1,191 @@
+# Agent Framework Lab - Lightning
+
+**Agent Framework Lab Lightning** is a specialized package that integrates [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) with [Agent-lightning](https://github.com/microsoft/agent-lightning) to provide reinforcement learning (RL) training capabilities for AI agents.
+
+This package enables you to train and fine-tune agents using advanced RL algorithms from VERL (e.g., GRPO, PPO, Reinforce++) with support for distributed training, multi-GPU setups, and comprehensive monitoring. It also supports complex multi-turn agent interactions during training and optimization techniques like prompt optimization. See the [Agent-lightning documentation](https://microsoft.github.io/agent-lightning/stable/) for details.
+
+> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `lightning` extra to use this module.
+
+## Installation
+
+Install the `agent-framework-lab` package with Lightning dependencies:
+
+```bash
+pip install "agent-framework-lab[lightning]"
+```
+
+### Optional Dependencies
+
+```bash
+# For math-related training
+pip install -e ".[lightning,math]"
+
+# For tau2 benchmarking
+pip install -e ".[lightning,tau2]"
+```
+
+To prepare for RL training, you'll also need to install dependencies like PyTorch, Ray, and vLLM. See the [Agent-lightning setup instructions](https://microsoft.github.io/agent-lightning/stable/tutorials/installation/) for more details.
+
+## Usage Patterns
+
+The basic usage pattern follows these steps:
+
+1. **Prepare your dataset** as a list of samples (typically dictionaries)
+2. **Create an agent function** that processes samples and returns evaluation scores
+3. **Decorate with `@agentlightning.rollout`** to enable training
+4. **Configure and run training** with the `agentlightning.Trainer` class
+
+### Example Implementation
+
+```python
+from agent_framework.lab.lightning import AgentFrameworkTracer
+from agentlightning import rollout, Trainer, LLM, Dataset
+from agentlightning.algorithm.verl import VERL
+
+TaskType = Any
+
+@rollout
+async def math_agent(task: TaskType, llm: LLM) -> float:
+    """A function that solves a math problem and returns the evaluation score."""
+    async with (
+        MCPStdioTool(name="calculator", command="uvx", args=["mcp-server-calculator"]) as mcp_server,
+        Agent(
+            client=OpenAIChatClient(
+                model=llm.model,
+                api_key="your-api-key",
+                base_url=llm.endpoint,
+            ),
+            name="MathAgent",
+            instructions="Solve the math problem and output answer after ###",
+            temperature=llm.sampling_parameters.get("temperature", 0.0),
+        ) as agent,
+    ):
+        result = await agent.run(task["question"], tools=mcp_server)
+        # Your evaluation logic here...
+        return evaluation_score
+
+# Training configuration
+config = {
+    "data": {"train_batch_size": 8},
+    "trainer": {"total_epochs": 2, "n_gpus_per_node": 1},
+    # ... additional config
+}
+
+# Initialize agent-framework tracer to send telemetry data to agent-lightning's observability backend
+tracer = AgentFrameworkTracer()
+
+trainer = Trainer(algorithm=VERL(config), tracer=tracer, n_workers=2)
+# Both train_dataset and val_dataset are lists of TaskType
+trainer.fit(math_agent, train_dataset, val_data=val_dataset)
+```
+
+## Example 1: Training a Math Agent
+
+This example trains an agent that uses an MCP calculator tool to solve math problems. The dataset is a small subset from the [Calc-X](https://huggingface.co/datasets/MU-NLPC/Calc-X) dataset. The Agent-lightning team has also experimented with a similar agent using a larger dataset. See [this example](https://github.com/microsoft/agent-lightning/tree/a63197355cc23b5b235c49fe7c20b54f9d4ebcd2/examples/calc_x) for more details.
+
+Running this example requires a minimum of 40GB GPU memory. If you don't have enough GPU memory, you can use a smaller model like `Qwen2.5-0.5B-Instruct`, though the results won't be as good. To run the example:
+
+```bash
+cd samples
+# Run the ray cluster (see the troubleshooting section for more details)
+ray start --head --dashboard-host=0.0.0.0
+# Run the training script
+python train_math_agent.py
+```
+
+To debug the agent used in the example, you can run the script with the `--debug` flag:
+
+```bash
+python train_math_agent.py --debug
+```
+
+The training curve below shows results with Qwen2.5-1.5B-Instruct and GRPO. Validation accuracy increases from 10% to 35% in the first 8 steps, then begins to overfit.
+
+![Training Curve](./assets/train_math_agent.png)
+
+## Example 2: Training a Tau2 Agent
+
+This advanced example demonstrates training on complex multi-agent scenarios using the Tau2 benchmark. It features a multi-agent setup with an assistant agent and a user simulator agent, training the assistant while keeping the user simulator fixed. The example incorporates a multi-step workflow with tool usage and complex evaluation metrics. Currently, training uses the airline domain with a 50/50 split between training and validation data.
+
+Before running this example, please read the [agent-lightning-lab-tau2](../tau2/README.md) documentation and follow the setup instructions.
+
+To run the example:
+
+```bash
+# Set required environment variables
+export TAU2_DATA_DIR="/path/to/tau2/data"
+
+# Used for user simulator and LLM judge
+export OPENAI_BASE_URL="your-endpoint"
+export OPENAI_API_KEY="your-key"
+
+# Used for tracking on Weights & Biases
+export WANDB_API_KEY="your-key"
+
+# Run the ray cluster
+ray start --head --dashboard-host=0.0.0.0
+
+# Train the tau2 agent
+cd samples
+python samples/train_tau2_agent.py
+
+# Debug mode
+python samples/train_tau2_agent.py --debug
+```
+
+This example uses more advanced Agent-lightning features compared to the math example. It's based on the `LitAgent` class rather than the `@rollout` decorator and involves concepts like resources and agent filtering. We recommend reading the [Agent-lightning documentation](https://microsoft.github.io/agent-lightning/stable/) to learn more.
+
+Results with Qwen2.5-1.5B-Instruct and GRPO are shown below. Validation accuracy improves from 28% to 40% over 8 epochs.
+
+![Training Curve](./assets/train_tau2_agent.png)
+
+## Troubleshooting
+
+### Ray Connection Issues
+
+Agent-lightning uses VERL for RL training, which depends on Ray. To avoid issues, it's recommended to start Ray manually beforehand. If you encounter Ray startup problems:
+
+```bash
+# Stop existing Ray processes
+ray stop
+
+# Start Ray with debugging enabled
+env RAY_DEBUG=legacy HYDRA_FULL_ERROR=1 VLLM_USE_V1=1 ray start --head --dashboard-host=0.0.0.0
+```
+
+**Important**: Run Ray commands in the same directory as your training script. Set any required environment variables (`WANDB_API_KEY`, `HF_TOKEN`) before starting Ray.
+
+### GPU Memory Issues
+
+1. **Reduce `gpu_memory_utilization`** to <0.8
+2. **Enable FSDP offloading**:
+   ```python
+   "fsdp_config": {
+       "param_offload": True,
+       "optimizer_offload": True,
+   }
+   ```
+3. **Decrease batch sizes**:
+   - `train_batch_size`
+   - `ppo_mini_batch_size`
+   - `log_prob_micro_batch_size_per_gpu`
+
+### Agent Debugging
+
+Always test your agent before training:
+
+```bash
+# Use debug mode to validate agent behavior
+python your_training_script.py --debug
+
+# Check agent responses and evaluation logic
+# Ensure proper tool integration and result extraction
+```
+
+## Contributing
+
+This package is part of the Microsoft Agent Framework Lab. Please see the main repository for contribution guidelines.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

python/packages/lab/lightning/agent_framework_lab_lightning/__init__.py

@@ -0,0 +1,37 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""RL Module for Microsoft Agent Framework."""
+
+from __future__ import annotations
+
+import importlib.metadata
+
+from agent_framework.observability import enable_instrumentation
+from agentlightning.tracer import (
+    AgentOpsTracer,
+)
+
+try:
+    __version__ = importlib.metadata.version(__name__)
+except importlib.metadata.PackageNotFoundError:
+    __version__ = "0.0.0"  # Fallback for development mode
+
+
+class AgentFrameworkTracer(AgentOpsTracer):
+    """Tracer for Agent-framework.
+
+    Tracer that enables OpenTelemetry observability for the Agent-framework,
+    so that the traces are visible to Agent-lightning.
+    """
+
+    def init(self) -> None:
+        """Initialize the agent-framework-lab-lightning for training."""
+        enable_instrumentation()
+        super().init()
+
+    def teardown(self) -> None:
+        """Teardown the agent-framework-lab-lightning for training."""
+        super().teardown()
+
+
+__all__: list[str] = ["AgentFrameworkTracer"]

python/packages/lab/lightning/assets/train_math_agent.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab35a2bd18794a32b76437671ae7e5749992c8aa781030b51eca2e56acfb362d
+size 153014

python/packages/lab/lightning/assets/train_tau2_agent.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fc96ee605b8d8e154892161cb01a42338d7781f5181afb060dac99d36eeb2c0
+size 189488

python/packages/lab/lightning/samples/data/math/test.jsonl

@@ -0,0 +1,20 @@
+{"id": "svamp__chal-551", "question": "Robin has some packages of gum. There are 7 pieces in each package. Robin has 6 extra pieces of gum. In all the number of pieces of gums robin has is 41. How many packages does Robin have?", "chain": "<gadget id=\"calculator\">41 - 6</gadget>\n<output>35</output>\n\n<gadget id=\"calculator\">35 / 7</gadget>\n<output>5</output>\n\n<result>5</result>", "result": "5", "source": "calc"}
+{"id": "ape210k__00027150", "question": "2 meters of floral cloth, how much rice is left after 80% is used", "chain": "<gadget id=\"calculator\">80 / 100</gadget>\n<output>4/5 = around 0.8</output>\n\n<gadget id=\"calculator\">1 - (4/5)</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">2 * (1/5)</gadget>\n<output>2/5 = around 0.4</output>\n\n<result>2/5 = around 0.4</result>", "result": "2/5", "source": "calc"}
+{"id": "ape210k__00287396", "question": "There are two schools, A and B. School A has 525 students, and school B has 50 fewer students than school A. How many students are there in school B?", "chain": "<gadget id=\"calculator\">525 * 2</gadget>\n<output>1_050</output>\n\n<gadget id=\"calculator\">1_050 - 50</gadget>\n<output>1_000</output>\n\n<result>1_000</result>", "result": "1_000", "source": "calc"}
+{"id": "gsm8k__xtQ5d23fzgEAhUdB", "question": "If Mark weighs 150 pounds and Susan weighs 20 pounds less than Mark.  And their friend Bob weighs twice as much as Susan.  What is the average weight of the 3 friends?", "chain": "Susan weighs 150 pounds - 20 pounds = \n<gadget id=\"calculator\">150-20</gadget>\n<output>130</output>\n130 pounds.\nBob weighs 2 * 130 pounds = \n<gadget id=\"calculator\">2*130</gadget>\n<output>260</output>\n260 pounds.\nThe friends total weight is 150 + 130 + 260 pounds = \n<gadget id=\"calculator\">150+130+260</gadget>\n<output>540</output>\n540 pounds.\nThe friends' average weight is 540 pounds / 3 = \n<gadget id=\"calculator\">540/3</gadget>\n<output>180</output>\n180 pounds.\n\n<result>180</result>", "result": "180", "source": "calc"}
+{"id": "svamp__chal-741", "question": "He had a total of 40 saltwater animals in different aquariums. Each aquarium has 2 animals in it. How many aquariums did he have?", "chain": "<gadget id=\"calculator\">40 / 2</gadget>\n<output>20</output>\n\n<result>20</result>", "result": "20", "source": "calc"}
+{"id": "asdiv_a__nluds-0023", "question": "You have collected 7 crickets. How many more crickets do you need to collect to have 11 crickets?", "chain": "<gadget id=\"calculator\">11 - 7</gadget>\n<output>4</output>\n\n<result>4</result>", "result": "4", "source": "calc"}
+{"id": "gsm8k__0oOjz5Ub66DF4inZ", "question": "There are 6 trees in Chris's yard.  Ferdinand has half the number of trees that Chris has.  Harry has 5 more than twice the number of trees that Ferdinand has.  How many more trees are in Harry's yard than Ferdinand's yard?", "chain": "Ferdinand:6/2=\n<gadget id=\"calculator\">6/2</gadget>\n<output>3</output>\n3 trees.\nHarry:5+2(3)=5+6=11 trees\n11-3=\n<gadget id=\"calculator\">11-3</gadget>\n<output>8</output>\n8 trees.\n\n<result>8</result>", "result": "8", "source": "calc"}
+{"id": "ape210k__00565195", "question": "During the May 1st period, Xiaoqiang\u2019s family went on a trip to other places. The planned consumption was 2,000 yuan, but the actual consumption was 1,800 yuan. How much less was the actual consumption than the plan?", "chain": "<gadget id=\"calculator\">2_000 - 1_800</gadget>\n<output>200</output>\n\n<gadget id=\"calculator\">200 / 2_000</gadget>\n<output>1/10 = around 0.1</output>\n\n<result>1/10 = around 0.1</result>", "result": "1/10", "source": "calc"}
+{"id": "mawps__E0wRRdRDwTmdqH2u", "question": "Milton had 238 peach. William clasped some peach. Now Milton has 51  peach. How many did William claspeds?", "chain": "<gadget id=\"calculator\">238 - 51</gadget>\n<output>187</output>\n\n<result>187</result>", "result": "187", "source": "calc"}
+{"id": "asdiv_a__nluds-0318", "question": "The map led them through the forest and into a cave. To open the cave doors, they need to put weights on the switch. If the switch already has 234 lbs. of weights and the total needed is 712 lbs.,, how much more weight to they need to add?", "chain": "<gadget id=\"calculator\">712 - 234</gadget>\n<output>478</output>\n\n<result>478</result>", "result": "478", "source": "calc"}
+{"id": "ape210k__00965281", "question": "The annual interest rate of the five-year national debt is 2.75%. If a person buys a national debt of 20,000 yuan, what is the total amount of principal and interest after maturity?", "chain": "<gadget id=\"calculator\">2.75 / 100</gadget>\n<output>0.0275</output>\n\n<gadget id=\"calculator\">20_000 * 0.0275 * 5</gadget>\n<output>2_750</output>\n\n<gadget id=\"calculator\">20_000 + 2_750</gadget>\n<output>22_750</output>\n\n<result>22_750</result>", "result": "22_750", "source": "calc"}
+{"id": "svamp__chal-289", "question": "Jack received 6 emails in the morning and 8 emails in the afternoon. How many more emails did Jack receive in the afternoon than in the morning?", "chain": "<gadget id=\"calculator\">8 - 6</gadget>\n<output>2</output>\n\n<result>2</result>", "result": "2", "source": "calc"}
+{"id": "ape210k__00829979", "question": "The fifth grade students participate in the big break exercise, and 12 people or 18 people can be divided into a row. If the number of students is less than 200, how many students can participate in the big break exercise this time?", "chain": "<gadget id=\"calculator\">36 * 5</gadget>\n<output>180</output>\n\n<result>180</result>", "result": "180", "source": "calc"}
+{"id": "ape210k__00909867", "question": "A and B process 1200 parts at the same time, and the plan is to complete it in 6 hours. A processes 80 parts per hour. To complete the work on time, how many parts does B need to process per hour? (column equations to solve problems)", "chain": "<gadget id=\"calculator\">80 * 6</gadget>\n<output>480</output>\n\n<gadget id=\"calculator\">1_200 - 480</gadget>\n<output>720</output>\n\n<gadget id=\"calculator\">720 / 6</gadget>\n<output>120</output>\n\n<result>120</result>", "result": "120", "source": "calc"}
+{"id": "svamp__chal-972", "question": "A mailman has to give 4 pieces of junk mail to each house in each of the 16 blocks. If there are 17 houses in each block, how many pieces of junk mail should he give in total?", "chain": "<gadget id=\"calculator\">4 * 17</gadget>\n<output>68</output>\n\n<gadget id=\"calculator\">68 * 16</gadget>\n<output>1_088</output>\n\n<result>1088</result>", "result": "1_088", "source": "calc"}
+{"id": "aqua_rat__j7vMuYEEajqH6GTH", "question": "5 horses are in a race. Mr.Jain selects two of horses at random and bets on them. The probability that he selected the winning horse is  Choose the correct choice:  A) 1/5  B) 2/5  C) 3/5  D) 4/5  E) 6/5", "chain": "There are 5 horses.   Probability of winning for each horse = 1/5.   Probability of winning with 2 selected horses= (1/5)+(1/5)= 2/5.   Answer is 2/5.   ANSWER:2/5\n<result>B</result>", "result": "B", "source": "calc"}
+{"id": "asdiv_a__nluds-0263", "question": "Feeling good about what he did, Mr. Anderson decided to continue giving to others. He went around the city and gave clothes to homeless people. If he gave 589 shirts and 345 trousers,, how many pieces of clothing did he gave out in total?", "chain": "<gadget id=\"calculator\">589 + 345</gadget>\n<output>934</output>\n\n<result>934</result>", "result": "934", "source": "calc"}
+{"id": "svamp__chal-968", "question": "Mary is baking a cake. The recipe calls for 10 cups of flour 2 cups of sugar and 80 cups of salt. She already put in 7 cups of flour. How many more cups of flour than cups of sugar does she need to add now?", "chain": "<gadget id=\"calculator\">10 - 7</gadget>\n<output>3</output>\n\n<gadget id=\"calculator\">3 - 2</gadget>\n<output>1</output>\n\n<result>1</result>", "result": "1", "source": "calc"}
+{"id": "gsm8k__aIzJoU5IRgriERup", "question": "A tub of ice cream costing $13 is now sold at $11. A packet of milk was sold at a discount of $0.5. How much will you save if you buy 2 tubs of ice cream and 4 packets of milk?", "chain": "The discount for each tub of ice cream is $13 - $11 = $\n<gadget id=\"calculator\">13-11</gadget>\n<output>2</output>\n2.\nSo the discount for 2 tubs of ice cream is $2 x 2 = $\n<gadget id=\"calculator\">2*2</gadget>\n<output>4</output>\n4.\nThe total discount for 4 packets of milk is $0.5 x 4 = $\n<gadget id=\"calculator\">0.5*4</gadget>\n<output>2</output>\n2.\nYou will save $4 + $2 = $6 for 2 tubs of ice cream and 4 packets of milk.\n\n<result>6</result>", "result": "6", "source": "calc"}
+{"id": "ape210k__00623575", "question": "In the art group, boys are girls (4/5), how much less boys than girls.", "chain": "<gadget id=\"calculator\">4 / 5</gadget>\n<output>4/5 = around 0.8</output>\n\n<gadget id=\"calculator\">1 - (4/5)</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">(1/5) / 1</gadget>\n<output>1/5 = around 0.2</output>\n\n<result>1/5 = around 0.2</result>", "result": "1/5", "source": "calc"}

python/packages/lab/lightning/samples/data/math/train.jsonl

@@ -0,0 +1,64 @@
+{"id": "ape210k__00384263", "question": "6.6 minus x (3/2) times equals 5.6.", "chain": "<gadget id=\"calculator\">6.6 - 5.6</gadget>\n<output>1</output>\n\n<gadget id=\"calculator\">3 / 2</gadget>\n<output>3/2 = around 1.5</output>\n\n<gadget id=\"calculator\">1 / (3/2)</gadget>\n<output>2/3 = around 0.666667</output>\n\n<result>2/3 = around 0.666667</result>", "result": "2/3", "source": "calc"}
+{"id": "ape210k__00469689", "question": "How many degrees of 75\u00b0 can form a right angle?", "chain": "<gadget id=\"calculator\">90 - 75</gadget>\n<output>15</output>\n\n<result>15</result>", "result": "15", "source": "calc"}
+{"id": "ape210k__00352031", "question": "Wang Li puts a piece of cake on the left side of the balance, and (1/4) piece of cake and a weight of 90 grams on the right side. At this time, the balance is just balanced. How much does a whole cake weigh in grams?", "chain": "<gadget id=\"calculator\">1 / 4</gadget>\n<output>1/4 = around 0.25</output>\n\n<gadget id=\"calculator\">90 / (1/4)</gadget>\n<output>360</output>\n\n<result>360</result>", "result": "360", "source": "calc"}
+{"id": "ape210k__00569876", "question": "Column calculation: Subtract 30% of (2/5) from 1, divide the difference by (11/50), what is the quotient?", "chain": "<gadget id=\"calculator\">2 / 5</gadget>\n<output>2/5 = around 0.4</output>\n\n<gadget id=\"calculator\">30 / 100</gadget>\n<output>3/10 = around 0.3</output>\n\n<gadget id=\"calculator\">(2/5) * (3/10)</gadget>\n<output>3/25 = around 0.12</output>\n\n<gadget id=\"calculator\">1 - (3/25)</gadget>\n<output>22/25 = around 0.88</output>\n\n<gadget id=\"calculator\">11 / 50</gadget>\n<output>11/50 = around 0.22</output>\n\n<gadget id=\"calculator\">(22/25) / (11/50)</gadget>\n<output>4</output>\n\n<result>4</result>", "result": "4", "source": "calc"}
+{"id": "ape210k__00767581", "question": "It takes Xiaofang 12 seconds to walk from the first floor to the third floor. At this speed, how many seconds does it take her to go from the third floor to the seventh floor?", "chain": "<gadget id=\"calculator\">3 - 1</gadget>\n<output>2</output>\n\n<gadget id=\"calculator\">12 / 2</gadget>\n<output>6</output>\n\n<gadget id=\"calculator\">7 - 3</gadget>\n<output>4</output>\n\n<gadget id=\"calculator\">6 * 4</gadget>\n<output>24</output>\n\n<result>24</result>", "result": "24", "source": "calc"}
+{"id": "aqua_rat__0bgRP2fAiH8URR9A", "question": "36 men can complete a piece of work in 18 days. In how many days will 108 men complete the same work ?\nChoose the correct choice\nA) 24 B) 77 C) 6 D) 29 E) 21", "chain": "Explanation:   Less Men, means more Days {Indirect Proportion}   Let the number of days be x   then,   108 : 36 :: 18 : x   x = 6   Answer: 6) 6 days\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "ape210k__00469555", "question": "The average score of Li Hong's Chinese unit test in the first three units of this semester is 92 points, and the average score of the first two units is 93 points. What is the score of the third language test?", "chain": "<gadget id=\"calculator\">92 * 3</gadget>\n<output>276</output>\n\n<gadget id=\"calculator\">93 * 2</gadget>\n<output>186</output>\n\n<gadget id=\"calculator\">276 - 186</gadget>\n<output>90</output>\n\n<result>90</result>", "result": "90", "source": "calc"}
+{"id": "aqua_rat__r54xvzEL3O9nU60t", "question": "Barbata invests $2400 in the National Bank at 5%. How much additional money must she invest at 10% so that the total annual income will be equal to 6% of her entire investment?\nPick one:\nA) 120\nB) 600\nC) 1000\nD) 360\nE) 240", "chain": "Let the additional invested amount for 10% interest be x;   Equation will be;   2400+0.05*2400+x+0.10x = 2400+x+0.06(2400+x)   0.05*2400+0.10x = 0.06x+0.06*2400   0.04x = 2400(0.06-0.05)   x = 2400*0.01/0.04= \n<gadget id=\"calculator\">2400*0.01/0.04</gadget>\n<output>600</output>\n600   Ans:600\n<result>B</result>", "result": "B", "source": "calc"}
+{"id": "ape210k__00851767", "question": "There are 30 boys in the dance group, and there are fewer girls than boys (1/5). How many girls are there?", "chain": "<gadget id=\"calculator\">1 / 5</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">1 - (1/5)</gadget>\n<output>4/5 = around 0.8</output>\n\n<gadget id=\"calculator\">30 * (4/5)</gadget>\n<output>24</output>\n\n<result>24</result>", "result": "24", "source": "calc"}
+{"id": "ape210k__00935567", "question": "Uncle Wang has 20 goats, 14 sheep, and 408 rabbits. How many times more rabbits are there than sheep?", "chain": "<gadget id=\"calculator\">20 + 14</gadget>\n<output>34</output>\n\n<gadget id=\"calculator\">408 / 34</gadget>\n<output>12</output>\n\n<result>12</result>", "result": "12", "source": "calc"}
+{"id": "math_qa__tyaZVO6Q2uEE7wBw", "question": "How much greater is the combined area in square inches of the front and back of a rectangular sheet of paper measuring 11 inches by 11 inches than that of a rectangular sheet of paper measuring 5.5 inches by 11 inches?\tChoose the correct choice from the following choices:\nA) 50 % B) 87 % C) 100 % D) 187 % E) 200 %", "chain": "<gadget id=\"calculator\">11 * 11</gadget>\n<output>121</output>\n\n<gadget id=\"calculator\">121 * 2</gadget>\n<output>242</output>\n\n<gadget id=\"calculator\">5.5 * 11</gadget>\n<output>60.5</output>\n\n<gadget id=\"calculator\">60.5 * 2</gadget>\n<output>121</output>\n\n<gadget id=\"calculator\">242 - 121</gadget>\n<output>121</output>\n\n<gadget id=\"calculator\">121 / 121</gadget>\n<output>1</output>\n\n<gadget id=\"calculator\">1 * 100</gadget>\n<output>100</output>\n\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "aqua_rat__Qtp9RDyp7cecUmpb", "question": "0.01 is what percent of 0.1?  Choices:  A) 1%  B) 10%  C) 100%  D) 50%  E) 25%", "chain": "Required percentage = 0.01*100/0.1 = 100/10= \n<gadget id=\"calculator\">100/10</gadget>\n<output>10</output>\n10%   Answer is 10%\n<result>B</result>", "result": "B", "source": "calc"}
+{"id": "ape210k__01121109", "question": "It is known that the sum of 9 consecutive natural numbers is 315, so what is the largest number among them.", "chain": "<gadget id=\"calculator\">315 / 9</gadget>\n<output>35</output>\n\n<gadget id=\"calculator\">35 + 1 + 1 + 1 + 1</gadget>\n<output>39</output>\n\n<result>39</result>", "result": "39", "source": "calc"}
+{"id": "aqua_rat__JwrYawp1nZkf5o7a", "question": "Josh spends a total of $5.5 buying S items in the convenience store. If each of the items is either a 5 cents single bubblegum, or a 50 cents bubblegum pack, then S may be which of the following?\nChoose the correct choice from the following answers.\nA) 99\nB) 100\nC) 101\nD) 112\nE) 113", "chain": "S items in the convenience store$5.5 = 550 cents   550 = 50a + 5b   =&gt;110 = 10a + b   b = 110 - 10a = 10(11-a)   Hence b is even and multiple of 10.   Possible values of b:   b = 10,20,30,40,50,60,70,80,90,100   a = 11,9,8,7,6,5,4,3,2,1   The total (a+b) is 21,29,38,47,56,65,74,83,92,101   The only option is 101. Hence 101.\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "ape210k__00348953", "question": "The road repair team repaired a road, 185 meters a day. It has been repaired for 20 days, and another 128 meters will be completed. How long is this road?", "chain": "<gadget id=\"calculator\">185 * 20</gadget>\n<output>3_700</output>\n\n<gadget id=\"calculator\">3_700 + 128</gadget>\n<output>3_828</output>\n\n<result>3_828</result>", "result": "3_828", "source": "calc"}
+{"id": "ape210k__00285692", "question": "The lateral expansion of a cylinder is a square with side length 8 cm. What is the lateral area of this cylinder in cm**2.", "chain": "<gadget id=\"calculator\">8 ** 2</gadget>\n<output>64</output>\n\n<result>64</result>", "result": "64", "source": "calc"}
+{"id": "ape210k__00391316", "question": "The greatest common factor of two numbers A and B is 5, and the least common multiple is 60. Where the number of A is 15, what is the number of B?", "chain": "<gadget id=\"calculator\">4 * 5</gadget>\n<output>20</output>\n\n<result>20</result>", "result": "20", "source": "calc"}
+{"id": "aqua_rat__yG3x6Th3XteHm4gg", "question": "The probability is 1/2 that a certain coin turns up heads on any given toss. If the coin is tossed five times, what is the probability that the coin turns up tails on at least one of the tosses? Choose the most appropriate option.\nA) 7/8 B) 15/16 C) 31/32 D) 21/32 E) 31/64", "chain": "P(5 heads)= 1/2*1/2*1/2*1/2*1/2=1/32.   P(at least one tail)=1-1/32=31/32.   The answer is 31/32.\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "ape210k__00459584", "question": "Xiaohua took a photo and wanted to make a wooden photo frame for the photo. The length of the photo frame is 25 cm and the width is 20 cm. At least how many centimeters of wooden strips should be prepared?", "chain": "<gadget id=\"calculator\">25 + 20</gadget>\n<output>45</output>\n\n<gadget id=\"calculator\">45 * 2</gadget>\n<output>90</output>\n\n<result>90</result>", "result": "90", "source": "calc"}
+{"id": "aqua_rat__eDeVHpDSC7yeRy8K", "question": "Two cards are drawn at random from a pack of 52 cards.what is the probability that either both are black or both are queen\nChoose the correct choice from the following\nA) 44/221\nB) 55/221\nC) 76/221\nD) 45/221\nE) 63/221", "chain": "WE HAVE N(S)=52C2=(52*51)/(2*1)= \n<gadget id=\"calculator\">(52*51)/(2*1)</gadget>\n<output>1_326</output>\n1326.   LET A=EVENT OF GETTING 55/221OTH 55/221LACK CARDS   55/221=EVENT OF GETTING 55/221OTH QUEENS   A\uf0c755/221=EVENT OF GETTING QUEEN OF 55/221LACK CARDS   N(A)=26C2=(26*25)/(2*1)= \n<gadget id=\"calculator\">(26*25)/(2*1)</gadget>\n<output>325</output>\n325,   N(55/221)=4C2=(4*3)/(2*1)= \n<gadget id=\"calculator\">(4*3)/(2*1)</gadget>\n<output>6</output>\n6 AND   N(A\uf0c755/221)=2C2=1   P(A)=N(A)/N(S)=325/1326;   P(55/221)=N(55/221)/N(S)=6/1326 AND   P(A\uf0c755/221)=N(A\uf0c755/221)/N(S)=1/1326   P(A\uf0c855/221)=P(A)+P(55/221)-P(A\uf0c755/221)=(325+6-1/1326)=330/1326=55/221   Option: 55/221\n<result>B</result>", "result": "B", "source": "calc"}
+{"id": "aqua_rat__8P5OAZaXVQlL4d8P", "question": "If two numbers are in the ratio 2:3. If 5 is added to both of the numbers then the ratio becomes 3:4 then find the smallest number?\nAnswers: A) A)10 B) B)18 C) C)20 D) D)24 E) E)26", "chain": "2:3   2x + 5 : 3x + 5 = 3 : 4   4[2x + 5] = 3[3x + 5]   8x + 20 = 9x + 15   9x - 8x = 20 - 15= \n<gadget id=\"calculator\">20 - 15</gadget>\n<output>5</output>\n5   Then smallest number is= 2   2x = 10   Correct Option 10\n<result>A</result>", "result": "A", "source": "calc"}
+{"id": "aqua_rat__1sGqyWbPyIDgCvSg", "question": "If a, b, c, d, e and f are integers and (ab + cdef) < 0, then what is the maximum number S of integers that can be negative? Choose the correct choice from the following answers.\nA) 2  B) 3  C) 4  D) 5  E) 6", "chain": "Minimuum should be 1   Maximum should be 4:   1 out of a or b to make the multiplication negative   3 out of c, d, e or f to make the multiplication negative.   Negative+Negative&lt;0   Answer:C   maximum will be 5..   you dont require both the multiplicatin to be negative for entire equation to be negative...   any one a or b can be negative to make ab negative and it can still be more(away from 0) than the multiplication of 4 other -ve numbers...   actually by writing minimum required as 1 out of 6,you are actually meaning S= 5 out of 6 also possible as you will see 5 or 1 will give you same equation..   ans 5\n<result>D</result>", "result": "D", "source": "calc"}
+{"id": "aqua_rat__mGakVxdUhicX5FmA", "question": "Find the area of the quadrilateral of one of its diagonals is 10 cm and its off sets 7 cm and 3 cm?  Choose the correct choice from the following answers\nA) 50 cm2\tB) 100 cm2\tC) 150 cm2\tD) 200 cm2\tE) 250 cm2", "chain": "1/2 * 10(7 + 3)   = 50 cm2   50 cm2nswer: 50 cm2\n<result>A</result>", "result": "A", "source": "calc"}
+{"id": "ape210k__00398657", "question": "A right-angled trapezoid has an upper base of 2 cm and a waist length of 10 cm. If the upper base is increased by 6 cm, it becomes a square. What is the perimeter of the trapezoid in cm?", "chain": "<gadget id=\"calculator\">2 + 6</gadget>\n<output>8</output>\n\n<gadget id=\"calculator\">8 * 2</gadget>\n<output>16</output>\n\n<gadget id=\"calculator\">2 + 16 + 10</gadget>\n<output>28</output>\n\n<result>28</result>", "result": "28", "source": "calc"}
+{"id": "aqua_rat__1I3XjjMFYW6ivEn6", "question": "In 1998 the profits of company N were 10 percent of revenues. In 1999, the revenues of company N fell by 20 percent, but profits were 14 percent of revenues. The profits in 1999 were what percent of the profits in 1998?  Answers.\nA) 80%  B) 105%  C) 120%  D) 112%  E) 138%", "chain": "0,112R = x/100*0.1R   Answer 112%\n<result>D</result>", "result": "D", "source": "calc"}
+{"id": "aqua_rat__orCiKDobdncZcRw8", "question": "In a market, a dozen eggs cost as much as a pound of rice, and a half-liter of kerosene costs as much as 6 eggs. If the cost of each pound of rice is $0.24, then how many cents does a liter of kerosene cost? [One dollar has 100 cents.]\nChoose the correct choice.\nA) 0.20 B) 0.24 C) 20 D) 24 E) 55", "chain": "A dozen eggs cost as much as a pound of rice --&gt; 12 eggs = 1 pound of rice = 24 cents;   A half-liter of kerosene costs as much as 6 eggs --&gt; 6 eggs = 1/2 liters of kerosene.   How many cents does a liter of kerosene cost --&gt; 1 liter of kerosene = 12 eggs = 12/12*24= \n<gadget id=\"calculator\">12/12*24</gadget>\n<output>24</output>\n24 cents.   Answer: 24.\n<result>D</result>", "result": "D", "source": "calc"}
+{"id": "ape210k__00022661", "question": "A company saved an average of 9 tons of water per day in the second quarter of last year. How many tons of water was saved in the second quarter?", "chain": "<gadget id=\"calculator\">30 + 31 + 30</gadget>\n<output>91</output>\n\n<gadget id=\"calculator\">91 * 9</gadget>\n<output>819</output>\n\n<result>819</result>", "result": "819", "source": "calc"}
+{"id": "ape210k__00154447", "question": "(2/10) meters are used up for a piece of iron wire, and (8/10) meters are left, what is the original length of this iron wire", "chain": "<gadget id=\"calculator\">2 / 10</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">8 / 10</gadget>\n<output>4/5 = around 0.8</output>\n\n<gadget id=\"calculator\">(1/5) + (4/5)</gadget>\n<output>1</output>\n\n<result>1</result>", "result": "1", "source": "calc"}
+{"id": "ape210k__00461545", "question": "Subtract 8 continuously from 496, and it will be 0 after how many times of subtraction.", "chain": "<gadget id=\"calculator\">496 / 8</gadget>\n<output>62</output>\n\n<result>62</result>", "result": "62", "source": "calc"}
+{"id": "ape210k__00220514", "question": "There are 54 cards in a deck of poker, ask: at least how many cards can be drawn from it to ensure that there are cards of four suits.", "chain": "<gadget id=\"calculator\">13 * 3</gadget>\n<output>39</output>\n\n<gadget id=\"calculator\">39 + 2 + 1</gadget>\n<output>42</output>\n\n<result>42</result>", "result": "42", "source": "calc"}
+{"id": "ape210k__00832821", "question": "There are several small balls of 4 different colors in the cloth bag, and the minimum number of small balls taken out can ensure that there must be 2 small balls of the same color.", "chain": "<gadget id=\"calculator\">4 + 1</gadget>\n<output>5</output>\n\n<result>5</result>", "result": "5", "source": "calc"}
+{"id": "ape210k__01121969", "question": "The fruit shop shipped 28 boxes of apples and pears each, each weighing 30 kg for pears and 25 kg for apples. How many kilograms of fruit does the fruit shop ship? (calculated in two ways)", "chain": "<gadget id=\"calculator\">28 * 30</gadget>\n<output>840</output>\n\n<gadget id=\"calculator\">28 * 25</gadget>\n<output>700</output>\n\n<gadget id=\"calculator\">840 + 700</gadget>\n<output>1_540</output>\n\n<result>1_540</result>", "result": "1_540", "source": "calc"}
+{"id": "ape210k__00910759", "question": "For a children's bicycle, the gear ratio of the front and rear gears is 12:7. If the rear gear rotates 24 times, how many times does the front gear rotate?", "chain": "<gadget id=\"calculator\">7 * 24</gadget>\n<output>168</output>\n\n<gadget id=\"calculator\">168 / 12</gadget>\n<output>14</output>\n\n<result>14</result>", "result": "14", "source": "calc"}
+{"id": "ape210k__01067071", "question": "If there is a basket of apples distributed among 6 people on average, there are 3 apples left. How many apples are there in this basket?", "chain": "<gadget id=\"calculator\">5 * 2 * 3</gadget>\n<output>30</output>\n\n<gadget id=\"calculator\">30 + 3</gadget>\n<output>33</output>\n\n<result>33</result>", "result": "33", "source": "calc"}
+{"id": "math_qa__nn3MNd29MSEsMQhG", "question": "Income and expenditure of a person are in the ratio 9 : 8. If the income of the person is Rs. 18000, then find his savings?\tChoose the correct choice:\nA) rs . 3600\nB) rs . 3603\nC) rs . 2000\nD) rs . 3632\nE) rs . 3602", "chain": "<gadget id=\"calculator\">8 / 9</gadget>\n<output>8/9 = around 0.888889</output>\n\n<gadget id=\"calculator\">(8/9) * 18_000</gadget>\n<output>16_000</output>\n\n<gadget id=\"calculator\">18_000 - 16_000</gadget>\n<output>2_000</output>\n\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "aqua_rat__ALWjr7wktcHDeMLJ", "question": "Twelve contestants at the county fair have entered their cakes to be judged in the cake decorating competition. A purple ribbon, blue ribbon, red ribbon, and white ribbon will be given to the first, second, third, and fourth place competitors, respectively. How many different ways are there to award the four ribbons to the contestants?\tAnswers\nA) 8!(4!*4!)\nB) 12!(8!*4!)\nC) 8!/4!\nD) 12!/8!\nE) 12!/4!", "chain": "The mistake you are doing is that you are neglecting the 4! ways in you can arrange 4 contestants for the 4 prizes.   Number of ways you can select 4 people out of 12 = 12C4   Once you select the 4 people, you have the following arrangement, PBRW (PBRW being the 4 prizes) but the same group of people can also be chosen against BRWP etc. Thus you get 4! ways of arranging 4 prizes.   Thus total possible ways = 12C4*4! = 12!/8!. 12!/8! is the correct answer.\n<result>D</result>", "result": "D", "source": "calc"}
+{"id": "ape210k__00947619", "question": "The material for the jacket is 1.55 meters per piece, and the material for the trousers is 1.05 meters per piece. How much rice cloth is needed to make a jacket and two trousers?", "chain": "<gadget id=\"calculator\">1.05 * 2</gadget>\n<output>2.1</output>\n\n<gadget id=\"calculator\">2.1 + 1.55</gadget>\n<output>3.65</output>\n\n<result>3.65</result>", "result": "3.65", "source": "calc"}
+{"id": "aqua_rat__cJf6UOtVHqOWRGHY", "question": "If m and n are positive integers of T such that m is a factor of n, how many positive multiples of m are less than or equal to 2n ?\tAnswers.\nA) 2m/n + 1  B) 2n/m + 1  C) 2n/(m+1)  D) 2m/n  E) 2n/m", "chain": "Lets say N=10, M=5   2N=20. so the answer should be 4 (20/5)   lets try to plug in the answers:   A-not an integer   B-not an integer   C-not an integer   D-1 (not the answer)   E-4 - the answer. (the only one).   I would choose E.   Method 2   N=M*A (A is an integer)   So - A=N/M   therefore in 2N A will be 2N/M   Again - Answer is 2n/m.\n<result>E</result>", "result": "E", "source": "calc"}
+{"id": "ape210k__00628803", "question": "The fruit shop shipped 450 kilograms of papaya, of which Taiwan papaya accounted for (2/9), how many kilograms of Taiwan papaya?", "chain": "<gadget id=\"calculator\">2 / 9</gadget>\n<output>2/9 = around 0.222222</output>\n\n<gadget id=\"calculator\">450 * (2/9)</gadget>\n<output>100</output>\n\n<result>100</result>", "result": "100", "source": "calc"}
+{"id": "ape210k__00708383", "question": "If the radius of a circle is 1 cm, what is the circumference of its semicircle in cm?", "chain": "<gadget id=\"calculator\">1 / 2</gadget>\n<output>1/2 = around 0.5</output>\n\n<gadget id=\"calculator\">2 * 3.14 * (1/2)</gadget>\n<output>3.14</output>\n\n<gadget id=\"calculator\">2 * 1</gadget>\n<output>2</output>\n\n<gadget id=\"calculator\">3.14 + 2</gadget>\n<output>5.14</output>\n\n<result>5.14</result>", "result": "5.14", "source": "calc"}
+{"id": "ape210k__00256542", "question": "(1/(1*3))+(1/(3*5))+(1/(5*7))+...(1/(47*49)).", "chain": "<gadget id=\"calculator\">1 / 49</gadget>\n<output>1/49 = around 0.020408</output>\n\n<gadget id=\"calculator\">1 - (1/49)</gadget>\n<output>48/49 = around 0.979592</output>\n\n<gadget id=\"calculator\">1 / 2</gadget>\n<output>1/2 = around 0.5</output>\n\n<gadget id=\"calculator\">(48/49) * (1/2)</gadget>\n<output>24/49 = around 0.489796</output>\n\n<result>24/49 = around 0.489796</result>", "result": "24/49", "source": "calc"}
+{"id": "ape210k__00046559", "question": "An oil barrel is filled with half a barrel of oil, and after pouring out (3/5) of the oil, there is still 8 kilograms of oil left. How many kilograms of oil can this oil barrel hold?", "chain": "<gadget id=\"calculator\">3 / 5</gadget>\n<output>3/5 = around 0.6</output>\n\n<gadget id=\"calculator\">1 - (3/5)</gadget>\n<output>2/5 = around 0.4</output>\n\n<gadget id=\"calculator\">8 / (2/5)</gadget>\n<output>20</output>\n\n<gadget id=\"calculator\">20 * 2</gadget>\n<output>40</output>\n\n<result>40</result>", "result": "40", "source": "calc"}
+{"id": "ape210k__00838787", "question": "cuboid plastic box containing liquid medicine has a length of 0.6 meters, a width of 0.25 meters, and a depth of 0.5 meters. If the whole box of medicines is packed into small bottles that can hold 400 milliliters, how many bottles should this box contain at least?", "chain": "<gadget id=\"calculator\">100 / 400</gadget>\n<output>1/4 = around 0.25</output>\n\n<gadget id=\"calculator\">0.6 * 100 * 0.25 * 100 * 0.5 * (1/4) * 10</gadget>\n<output>1_875</output>\n\n<result>1_875</result>", "result": "1_875", "source": "calc"}
+{"id": "ape210k__01031589", "question": "Stack 2,100 cubes with side lengths of 1 cm to form a solid cuboid. Its height is 10 cm, and its length and width are greater than its height. What is the sum of the length and width of this cuboid in cm?", "chain": "<gadget id=\"calculator\">15 + 14</gadget>\n<output>29</output>\n\n<result>29</result>", "result": "29", "source": "calc"}
+{"id": "aqua_rat__7f9Pgjdk9qlo5Tf9", "question": "If the product 4864*9 P 2 is divisible by 12, the value of p?\nOptions\nA) 1  B) 5  C) 6  D) 7  E) 9", "chain": "Explanation:   clearly 4864 is divisible by 4   So 9 P 2 must be divisible by 3.So(9+P+2) must be divisible by 3.   so P=1.   1nswer: 1) 1\n<result>A</result>", "result": "A", "source": "calc"}
+{"id": "ape210k__00563400", "question": "It is known that \u22201+\u22202=150\u00b0, \u22201=67\u00b0, then \u22202=how much.", "chain": "<gadget id=\"calculator\">150 - 67</gadget>\n<output>83</output>\n\n<result>83</result>", "result": "83", "source": "calc"}
+{"id": "ape210k__00850129", "question": "The weight of an astronaut on the earth is 72 kg, which is 6 times his weight on the moon. How many kilograms would he weigh on the moon?", "chain": "<gadget id=\"calculator\">72 / 6</gadget>\n<output>12</output>\n\n<result>12</result>", "result": "12", "source": "calc"}
+{"id": "ape210k__00838563", "question": "In a bag of sugar, toffee accounts for 25% of the total number. After putting in 20 pieces of fruit candy, toffee accounts for 20% of the total number. How many pieces of toffee are there?", "chain": "<gadget id=\"calculator\">25 / 100</gadget>\n<output>1/4 = around 0.25</output>\n\n<gadget id=\"calculator\">80 * (1/4)</gadget>\n<output>20</output>\n\n<result>20</result>", "result": "20", "source": "calc"}
+{"id": "ape210k__00921605", "question": "To process a batch of parts, A alone can complete it in 9 days, and B alone can complete it in 12 days.", "chain": "<gadget id=\"calculator\">1 / 9</gadget>\n<output>1/9 = around 0.111111</output>\n\n<gadget id=\"calculator\">1 / 12</gadget>\n<output>1/12 = around 0.083333</output>\n\n<gadget id=\"calculator\">(1/9) + (1/12)</gadget>\n<output>7/36 = around 0.194444</output>\n\n<gadget id=\"calculator\">1 / (7/36)</gadget>\n<output>36/7 = around 5.142857</output>\n\n<result>36/7 = around 5.142857</result>", "result": "36/7", "source": "calc"}
+{"id": "ape210k__01057303", "question": "A batch of coal is shipped from the boiler room of a certain factory. According to the daily consumption of 150 kg of coal in the old boiler, it can be used for 120 days. If a new boiler is used, the coal consumption will be reduced by 20%. How many more days can this batch of coal be burned?", "chain": "<gadget id=\"calculator\">20 / 100</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">1 - (1/5)</gadget>\n<output>4/5 = around 0.8</output>\n\n<gadget id=\"calculator\">150 * (4/5)</gadget>\n<output>120</output>\n\n<gadget id=\"calculator\">120 / 120</gadget>\n<output>1</output>\n\n<gadget id=\"calculator\">150 * 1</gadget>\n<output>150</output>\n\n<gadget id=\"calculator\">150 - 120</gadget>\n<output>30</output>\n\n<result>30</result>", "result": "30", "source": "calc"}
+{"id": "ape210k__00442282", "question": "Dissolve 1 gram of sugar in 10 grams of water, how much sugar accounts for the sugar water", "chain": "<gadget id=\"calculator\">1 + 10</gadget>\n<output>11</output>\n\n<gadget id=\"calculator\">1 / 11</gadget>\n<output>1/11 = around 0.090909</output>\n\n<result>1/11 = around 0.090909</result>", "result": "1/11", "source": "calc"}
+{"id": "ape210k__00030668", "question": "In the final exam, Dalang's average score in the four subjects was 90 before the English score was announced. After the English score was announced, his average score increased by 2 points. How many points did Dawa score in the English test?", "chain": "<gadget id=\"calculator\">90 + 2</gadget>\n<output>92</output>\n\n<gadget id=\"calculator\">92 * 5</gadget>\n<output>460</output>\n\n<gadget id=\"calculator\">90 * 4</gadget>\n<output>360</output>\n\n<gadget id=\"calculator\">460 - 360</gadget>\n<output>100</output>\n\n<result>100</result>", "result": "100", "source": "calc"}
+{"id": "ape210k__00295527", "question": "A project has been completed (2/3) by 8 workers in 24 days, and it needs to be completed in 28 days. (Continue to work) How many more people are needed?", "chain": "<gadget id=\"calculator\">2 / 3</gadget>\n<output>2/3 = around 0.666667</output>\n\n<gadget id=\"calculator\">1 - (2/3)</gadget>\n<output>1/3 = around 0.333333</output>\n\n<gadget id=\"calculator\">(2/3) / 8 / 24</gadget>\n<output>1/288 = around 0.003472</output>\n\n<gadget id=\"calculator\">28 - 24</gadget>\n<output>4</output>\n\n<gadget id=\"calculator\">(1/288) * 4</gadget>\n<output>1/72 = around 0.013889</output>\n\n<gadget id=\"calculator\">(1/3) / (1/72)</gadget>\n<output>24</output>\n\n<gadget id=\"calculator\">24 - 8</gadget>\n<output>16</output>\n\n<result>16</result>", "result": "16", "source": "calc"}
+{"id": "aqua_rat__tasT86j2f4Cpn4kL", "question": "750 students took the test on English and Maths. 35% students failed in english and 45% failed in maths. 40% of those who passed in maths also passed in english, then how many students failed in both ? Choose the correct choice:\nA) a) 162\nB) b) 15\nC) c) 60\nD) d) 38\nE) e) 12", "chain": "Passed in english = 65%   Passed in maths = 55%   Passed in both = 40% of 55% = 2/5 * (55%) = 22%   Passed in (English + Maths - b) 15oth + Neither) 2/5 * (55 )= \n<gadget id=\"calculator\">2/5 * (55 )</gadget>\n<output>22</output>\n22%   Passed in (English + Maths - b) 15oth + Neither)= 100%   65 + 55 - 22 + Neither = 100   Neither = 100 - 98= \n<gadget id=\"calculator\">100 - 98</gadget>\n<output>2</output>\n2%= 0.02 * 750= \n<gadget id=\"calculator\">0.02 * 750</gadget>\n<output>15</output>\n15   Answer: b) 15\n<result>B</result>", "result": "B", "source": "calc"}
+{"id": "aqua_rat__OeZ7ZLBAPlOfp6Wg", "question": "A batsman scored 130 runs which included 3 boundaries and 8 sixes. What percent of his total score did he make by running between the wickets? Choose the correct choice from the following answers.\nA) 45(4/11) %  B) 45 %  C) 53(11/13) %  D) 44(5/11) %  E) None of these", "chain": "Explanation :   Total runs scored = 130   Total runs scored from boundaries and sixes = 3 x 4 + 8 x 6 = 60   Total runs scored by running between the wickets = 130 - 60= \n<gadget id=\"calculator\">130 - 60</gadget>\n<output>70</output>\n70   Required %= (70/130) \u00d7 100 = 700/13 = 53(11/13)%   Answer : Option 53(11/13) %\n<result>C</result>", "result": "C", "source": "calc"}
+{"id": "ape210k__00084943", "question": "(19/20) minus (13/20), and how much to subtract, the difference is (1/5).", "chain": "<gadget id=\"calculator\">19 / 20</gadget>\n<output>19/20 = around 0.95</output>\n\n<gadget id=\"calculator\">13 / 20</gadget>\n<output>13/20 = around 0.65</output>\n\n<gadget id=\"calculator\">1 / 5</gadget>\n<output>1/5 = around 0.2</output>\n\n<gadget id=\"calculator\">(19/20) - (13/20) - (1/5)</gadget>\n<output>1/10 = around 0.1</output>\n\n<result>1/10 = around 0.1</result>", "result": "1/10", "source": "calc"}
+{"id": "ape210k__00010503", "question": "Add 8 trees equidistantly between the two trees, so that the distance between the first tree and the fifth tree is 20 meters, how many meters are the distance between the original two trees?", "chain": "<gadget id=\"calculator\">5 - 1</gadget>\n<output>4</output>\n\n<gadget id=\"calculator\">20 / 4</gadget>\n<output>5</output>\n\n<gadget id=\"calculator\">8 + 1</gadget>\n<output>9</output>\n\n<gadget id=\"calculator\">5 * 9</gadget>\n<output>45</output>\n\n<result>45</result>", "result": "45", "source": "calc"}
+{"id": "ape210k__00468021", "question": "A 2-meter-long iron wire uses up 4 decimeters, and the rest forms a square. What is the side length of the enclosed square?", "chain": "<gadget id=\"calculator\">2 * 10</gadget>\n<output>20</output>\n\n<gadget id=\"calculator\">20 - 4</gadget>\n<output>16</output>\n\n<gadget id=\"calculator\">16 / 4</gadget>\n<output>4</output>\n\n<result>4</result>", "result": "4", "source": "calc"}
+{"id": "ape210k__00041697", "question": "There is a rectangular vegetable field with a length of 35 meters and a width of 14 meters. What is the area of this vegetable field in square meters?", "chain": "<gadget id=\"calculator\">35 * 14</gadget>\n<output>490</output>\n\n<result>490</result>", "result": "490", "source": "calc"}
+{"id": "aqua_rat__sr81Rp7oIv2lWfLl", "question": "Rectangular tile each of size 80cm by 40cm must be laid horizontally on a rectangular floor of size 130cm by 230cm,such that the tiles do not overlap and they are placed with edges jutting against each other on all edges. A tile can be placed in any orientation so long as its edges are parallel to the edges of floor. No tile should overshoot any edge of the floor. The maximum number of tiles that can be accommodated on the floor is:  Choose the correct option:\nA) 6\tB) 2\tC) 8\tD) 9\tE) 7", "chain": "Area of tile = 80*40= \n<gadget id=\"calculator\">80*40</gadget>\n<output>3_200</output>\n3200   Area of floor= 130*230= \n<gadget id=\"calculator\">130*230</gadget>\n<output>29_900</output>\n29900   No of tiles= 29900/3200 = 9.34   So, the no of tile = 9   ANSWER:9\n<result>D</result>", "result": "D", "source": "calc"}
+{"id": "ape210k__00607118", "question": "Xiaoxiao scored 92 in Chinese, 98 in mathematics, and 95 in English in the final test. What is her average score in the three subjects?", "chain": "<gadget id=\"calculator\">92 + 98 + 95</gadget>\n<output>285</output>\n\n<gadget id=\"calculator\">285 / 3</gadget>\n<output>95</output>\n\n<result>95</result>", "result": "95", "source": "calc"}
+{"id": "ape210k__00940095", "question": "The fruit store shipped 425 kilograms of apples, and the pears shipped were 80 kilograms less than four times the apples. How many kilograms of pears did the fruit store ship?", "chain": "<gadget id=\"calculator\">425 * 4</gadget>\n<output>1_700</output>\n\n<gadget id=\"calculator\">1_700 - 80</gadget>\n<output>1_620</output>\n\n<result>1_620</result>", "result": "1_620", "source": "calc"}
+{"id": "ape210k__00428406", "question": "If a cuboid has a base area of 80 cm2 and a height of 7 cm, what is its volume in cubic cm?", "chain": "<gadget id=\"calculator\">80 * 7</gadget>\n<output>560</output>\n\n<result>560</result>", "result": "560", "source": "calc"}
+{"id": "math_qa__Oc10lU6MVDif6R2Q", "question": "Matt and Peter can do together a piece of work in 20 days. After they have worked together for 12 days Matt stops and Peter completes the remaining work in 8 days. In how many days Peter complete the work separately.\nChoices:\nA) 21\nB) 24\nC) 20\nD) 25\nE) 30", "chain": "<gadget id=\"calculator\">1 / 20</gadget>\n<output>1/20 = around 0.05</output>\n\n<gadget id=\"calculator\">12 * (1/20)</gadget>\n<output>3/5 = around 0.6</output>\n\n<gadget id=\"calculator\">1 - (3/5)</gadget>\n<output>2/5 = around 0.4</output>\n\n<gadget id=\"calculator\">8 / (2/5)</gadget>\n<output>20</output>\n\n<result>C</result>", "result": "C", "source": "calc"}

python/packages/lab/lightning/samples/train_math_agent.py

@@ -0,0 +1,331 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""This sample demonstrates the basic usage pattern of agent-framework-lab-lightning.
+
+It trains a math agent using a dataset in `data/math/` to solve mathematical problems
+using an MCP calculator tool.
+
+One GPU with 40GB of memory is sufficient for this sample.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import math
+import os
+import re
+import string
+from typing import TypedDict, cast
+
+import sympy  # type: ignore[import-untyped,reportMissingImports]
+from agent_framework import Agent, AgentResponse, MCPStdioTool
+from agent_framework.lab.lightning import AgentFrameworkTracer
+from agent_framework.openai import OpenAIChatClient
+from agentlightning import LLM, Dataset, Trainer, rollout
+from agentlightning.algorithm.verl import VERL
+
+
+class MathProblem(TypedDict):
+    """This TypedDict defines the structure of each training sample.
+
+    Your task structure should contain all the information needed for:
+
+    - The agent to process the task (e.g., 'question')
+    - Evaluation (e.g., 'result' for ground truth)
+
+    This type is optional. Not necessary to make the example work.
+    """
+
+    # The fields come from the dataset
+    id: str
+    question: str  # The math problem for the agent to solve
+    chain: str  # Step-by-step solution (not used in training)
+    result: str  # Ground truth answer for evaluation
+    source: str
+
+
+def _load_jsonl(file_path: str) -> Dataset[MathProblem]:
+    """Load your dataset as a list of task samples.
+
+    Each sample should match your task structure (MathProblem in this case).
+    """
+    with open(file_path) as f:
+        raw_data = [MathProblem(**json.loads(line)) for line in f]
+    return cast(Dataset[MathProblem], raw_data)
+
+
+# Evaluation logic
+# These functions evaluate whether the agent's answer matches the ground truth.
+# Robust evaluation is crucial for RL training - the reward signal guides learning.
+
+
+def _normalize_option(option: str) -> str:
+    return re.sub(r"(\s+|\(|\))", "", option)
+
+
+def _is_option_result(result: str) -> bool:
+    return _normalize_option(result) in list(string.ascii_letters)
+
+
+def _float_eval(input_str: str) -> float:
+    if " = around " in input_str:
+        input_str = input_str.split(" = around ")[0]
+    expr = sympy.parse_expr(input_str, evaluate=True)
+    return float(expr.evalf())
+
+
+def _scalar_are_results_same(pred_result: str, true_result: str, rel_tol: float) -> bool:
+    pred_result = str(pred_result) if pred_result is not None else ""
+    true_result = str(true_result) if true_result is not None else ""
+
+    if pred_result.strip() == true_result.strip():
+        return True
+
+    if _is_option_result(true_result):
+        # The task is to select correct option
+        true_result = _normalize_option(true_result)
+        pred_result = _normalize_option(pred_result)
+        return pred_result == true_result
+
+    # The task is to calculate the result as a number
+    try:
+        pred_float = _float_eval(pred_result)
+        true_float = _float_eval(true_result)
+        return math.isclose(pred_float, true_float, rel_tol=rel_tol)
+    except Exception:  # noqa: S110
+        pass
+
+    return False
+
+
+def _is_result_correct(prediction: str, ground_truth: str) -> float:
+    return float(_scalar_are_results_same(prediction, ground_truth, 1e-2))
+
+
+def evaluate(result: AgentResponse, ground_truth: str) -> float:
+    """Main evaluation function that extracts the agent's answer and compares with ground truth.
+
+    This function:
+    1. Extracts the final answer from the agent's response (after ###)
+    2. Compares it with the ground truth using mathematical equivalence
+    3. Returns a reward score (0.0 or 1.0) for RL training
+
+    The reward signal is critical - it directly influences what the model learns.
+    """
+    # Check if agent provided any response
+    if len(result.messages) == 0:
+        print("No response from agent. Assuming incorrect.")
+        return 0.0
+    final_message = result.messages[-1].text
+
+    # Extract answer after ### marker (as specified in agent instructions)
+    answer = re.search(r"###\s*(.+?)(\s*###|$)", final_message)
+    if answer is None:
+        print("No answer can be extracted from agent's response. Assuming incorrect.")
+        return 0.0
+    answer = answer.group(1)
+
+    # Compare extracted answer with ground truth
+    reward = _is_result_correct(answer, ground_truth)
+    print(f"Reward: {reward}")
+    return reward
+
+
+# Agent Logic
+
+# Clear instructions are important for consistent agent behavior
+# The ### format helps with reliable answer extraction during evaluation
+AGENT_INSTRUCTION = """
+Solve the following math problem. Use the calculator tool to help you calculate math expressions.
+
+Output the answer when you are ready. The answer should be after three sharps (`###`),
+with no extra punctuations or texts. For example: ### 123
+""".strip()
+
+
+# The @rollout decorator is the key integration point with agent-lightning.
+# It tells the training system that this function defines a trainable agent.
+@rollout
+async def math_agent(task: MathProblem, llm: LLM) -> float:
+    """This is your trainable agent function.
+
+    Key points:
+
+    1. Must be decorated with @rollout
+    2. Takes a task sample and LLM object as parameters
+    3. Returns a float reward score (0.0 to 1.0 typically)
+    4. The LLM object contains the model being trained and its configuration
+
+    During training:
+    - llm.model: The model checkpoint being trained
+    - llm.endpoint: vLLM server endpoint for inference
+    - llm.sampling_parameters: Temperature, etc.
+    """
+    # Create the Agent Framework components
+    # MCPStdioTool provides calculator functionality via MCP protocol
+    async with (
+        MCPStdioTool(name="calculator", command="uvx", args=["mcp-server-calculator"]) as mcp_server,
+        Agent(
+            client=OpenAIChatClient(
+                model=llm.model,  # This is the model being trained
+                api_key=os.getenv("OPENAI_API_KEY") or "dummy",  # Can be dummy when connecting to training LLM
+                base_url=llm.endpoint,  # vLLM server endpoint provided by agent-lightning
+            ),
+            name="MathAgent",
+            instructions=AGENT_INSTRUCTION,
+            temperature=llm.sampling_parameters.get("temperature", 0.0),
+        ) as agent,
+    ):
+        print(f"Task: {task['question'][:10]}...")
+        # Run the agent on the task
+        result = await agent.run(task["question"], tools=mcp_server)
+        print(f"Agent responses: {result}")
+
+        # Evaluate and return reward - this is what drives RL training
+        return evaluate(result, task["result"])
+
+
+def main():
+    """Main entrypoint."""
+    # Configure RL training
+    # This configuration controls all aspects of the RL training process.
+    # Key sections: algorithm, data, rollout, actor, trainer
+    rl_training_config = {
+        "algorithm": {
+            # Advantage estimator type: "gae", "grpo", "reinforce_plus_plus", etc.
+            "adv_estimator": "grpo"
+        },
+        "data": {
+            # Uses this many tasks from the dataset to perform rollouts
+            "train_batch_size": 8,
+            # Used to filter out the over-long prompt-response pairs
+            "max_prompt_length": 4096,
+            "max_response_length": 1024,
+        },
+        "actor_rollout_ref": {
+            # Controls the rollout process
+            "rollout": {
+                # Set to 1 unless you want to use TP in multiple GPUs
+                "tensor_model_parallel_size": 1,
+                # Repeat each task N many times. Required by G(rouped)RPO
+                "n": 4,
+                # Controls the batch size per GPU when computing the log-prob
+                "log_prob_micro_batch_size_per_gpu": 2,
+                # Controls the multi-turn format (this is binded to the LLM used)
+                # See https://docs.vllm.ai/en/stable/features/tool_calling.html
+                "multi_turn": {"format": "hermes"},
+                # Only vllm is supported for now
+                "name": "vllm",
+                # Controls the GPU memory utilization of vLLM
+                # You might want to set this to under 0.8 to prevent OOM
+                "gpu_memory_utilization": 0.7,
+            },
+            "actor": {
+                # Split each sample into sub-batches of this size for PPO
+                "ppo_mini_batch_size": 8,
+                # Local per-GPU micro batch size
+                "ppo_micro_batch_size_per_gpu": 2,
+                # Optimizer configuration
+                "optim": {"lr": 1e-6},
+                # Whether to use KL loss during training
+                "use_kl_loss": False,
+                # PPO clipping ratios for policy updates
+                "clip_ratio_low": 0.2,
+                "clip_ratio_high": 0.3,
+                # FSDP (Fully Sharded Data Parallel) configuration for memory efficiency
+                # Useful when you don't have enough GPU memory
+                "fsdp_config": {
+                    # Whether to offload parameters to CPU
+                    "param_offload": True,
+                    # Whether to offload optimizer state to CPU
+                    "optimizer_offload": True,
+                },
+            },
+            # Reference model config
+            "ref": {
+                # Controls the batch size per GPU when computing log-prob for reference model
+                "log_prob_micro_batch_size_per_gpu": 2,
+                "fsdp_config": {"param_offload": True},
+            },
+            # Common configs for the model
+            "model": {
+                # Huggingface model path.
+                # If you want to train a different model, change the path here.
+                "path": "Qwen/Qwen2.5-1.5B-Instruct",
+                # Whether to remove padding tokens in inputs during training
+                "use_remove_padding": True,
+                # Enable gradient checkpointing for memory efficiency
+                "enable_gradient_checkpointing": True,
+            },
+        },
+        # Config for the trainer
+        "trainer": {
+            # Number of GPUs per node
+            "n_gpus_per_node": 1,
+            # Whether to run validation before training begins
+            "val_before_train": True,
+            # Logging backends to use: "console", "wandb", etc.
+            "logger": ["console"],
+            # Number of nodes used in the training
+            "nnodes": 1,
+            # Validation frequency (in training iterations)
+            "test_freq": 4,
+            # Number of epochs in training
+            "total_epochs": 2,
+        },
+    }
+
+    # Load your datasets
+    train_dataset = _load_jsonl("data/math/train.jsonl")
+    val_dataset = _load_jsonl("data/math/test.jsonl")
+
+    # Preview the data to ensure it's loaded correctly
+    print("First 5 rows of train dataset:")
+    for i in range(5):
+        print(train_dataset[i])
+    print("First 5 rows of val dataset:")
+    for i in range(5):
+        print(val_dataset[i])
+
+    # Create trainer with VERL algorithm and start training
+    # n_workers: Number of rollout workers (processes) for parallel data collection
+    trainer = Trainer(algorithm=VERL(rl_training_config), tracer=AgentFrameworkTracer(), n_workers=2)
+
+    # This starts the actual RL training loop:
+    # 1. Collect rollouts using current model
+    # 2. Compute advantages and train the model
+    # 3. Repeat for specified number of epochs
+    trainer.fit(math_agent, train_dataset, val_dataset=val_dataset)
+
+
+def debug():
+    """Debug mode allows you to test your agent function before training.
+
+    Always run debug mode first before starting expensive RL training!
+    """
+    train_dataset = _load_jsonl("data/math/train.jsonl")
+    train_sample = train_dataset[0]
+
+    # Use a known good model for debugging (not the one being trained)
+    model = "gpt-4o-mini"
+    base_url = os.getenv("OPENAI_BASE_URL")
+    api_key = os.getenv("OPENAI_API_KEY")
+    if api_key is None:
+        raise ValueError("OPENAI_API_KEY must be set")
+    if base_url is None:
+        raise ValueError("OPENAI_BASE_URL must be set")
+
+    # Test your agent function with a sample task
+    asyncio.run(math_agent(train_sample, LLM(model=model, endpoint=base_url)))  # type: ignore
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--debug", action="store_true")
+    args = parser.parse_args()
+    if args.debug:
+        debug()
+    else:
+        main()

python/packages/lab/lightning/samples/train_tau2_agent.py

@@ -0,0 +1,231 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Advanced example showing multi-agent RL training using Tau2 benchmark.
+
+This demonstrates:
+- LitAgent class-based approach (vs @rollout decorator)
+- Multi-agent scenarios with agent filtering
+- Resource management for complex setups
+- Integration with external benchmarks
+
+Builds on concepts from train_math_agent.py with additional complexity.
+Requires one GPU of at least 80GB of memory.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import random
+import time
+import traceback
+from pathlib import Path
+from typing import TypedDict, cast
+
+from agent_framework.lab.lightning import AgentFrameworkTracer
+from agent_framework.lab.tau2 import ASSISTANT_AGENT_ID, patch_env_set_state  # type: ignore
+from agent_framework.lab.tau2 import TaskRunner as Tau2TaskRunner  # type: ignore
+from agent_framework.openai import OpenAIChatClient
+from agentlightning import LLM, Dataset, LitAgent, NamedResources, Rollout, Trainer
+from agentlightning.algorithm.verl import VERL
+from tau2.data_model.tasks import Task as Tau2Task  # type: ignore[import-untyped]
+
+
+# Tau2 tasks are complex objects that need special handling during distributed training
+class SerializedTask(TypedDict):
+    """Tau2 task object type."""
+
+    id: str
+    data: str  # JSON-serialized task data to prevent HuggingFace conversion issues
+
+
+def _load_dataset() -> tuple[Dataset[SerializedTask], Dataset[SerializedTask]]:
+    """Load and prepare Tau2 dataset with proper serialization.
+
+    It takes external data dependency (TAU2_DATA_DIR) and uses deterministic train/val split for reproducibility.
+    """
+    data_dir = os.getenv("TAU2_DATA_DIR")
+    if data_dir is None:
+        raise ValueError("TAU2_DATA_DIR must be set")
+    tasks_path = Path(data_dir) / "tau2/domains/airline/tasks.json"
+    with tasks_path.open("r") as f:
+        dataset = json.load(f)
+
+    # Serialize complex task objects to prevent HuggingFace tokenizer issues
+    dataset = [{"id": task["id"], "data": json.dumps(task)} for task in dataset]
+
+    # Deterministic train/val split (25/25) for reproducible experiments
+    random_state = random.Random(42)  # noqa: S311
+    indices = list(range(len(dataset)))
+    random_state.shuffle(indices)
+    train_indices = indices[: int(len(dataset) * 0.5)]
+    val_indices = indices[int(len(dataset) * 0.5) :]
+    print(f"Train indices: {train_indices}")
+    print(f"Val indices: {val_indices}")
+    train_dataset = [dataset[i] for i in train_indices]
+    val_dataset = [dataset[i] for i in val_indices]
+
+    return cast(Dataset[SerializedTask], train_dataset), cast(Dataset[SerializedTask], val_dataset)
+
+
+# Alternative to @rollout: LitAgent class for advanced scenarios
+# Use this approach when you need:
+# - Agent filtering (training only specific agents in multi-agent setup)
+# - Resource management (multiple LLMs, databases, etc.)
+# - Complex initialization logic
+class Tau2Agent(LitAgent):
+    """Class-based agent with advanced resource management and agent filtering."""
+
+    async def rollout_async(self, task: SerializedTask, resources: NamedResources, rollout: Rollout) -> float:
+        """The main rollout method. Similar to @rollout but with more control."""
+        llm = resources.get("main_llm")
+        if not isinstance(llm, LLM):
+            raise ValueError("main_llm must be an instance of LLM")
+
+        openai_base_url = os.getenv("OPENAI_BASE_URL")
+        openai_api_key = os.getenv("OPENAI_API_KEY")
+        if openai_base_url is None:
+            raise ValueError("OPENAI_BASE_URL must be set")
+        if openai_api_key is None:
+            raise ValueError("OPENAI_API_KEY must be set")
+
+        # Deserialize the complex task object
+        task_data = json.loads(task["data"])
+        task_obj = Tau2Task(**task_data)
+
+        # Multi-agent setup: assistant (trainable) + user simulator (fixed)
+        runner = Tau2TaskRunner(
+            max_steps=100,
+            assistant_window_size=4000,
+            assistant_sampling_temperature=llm.sampling_parameters.get("temperature", 0.0),
+        )
+
+        # Assistant agent: uses the model being trained
+        assistant_chat_client = OpenAIChatClient(
+            base_url=llm.endpoint,  # vLLM endpoint for the model being trained
+            api_key=openai_api_key,
+            model=llm.model,  # Model ID being trained
+        )
+
+        # User simulator: uses a fixed, capable model for consistent simulation
+        user_simulator_chat_client = OpenAIChatClient(
+            base_url=openai_base_url,  # External API endpoint
+            api_key=openai_api_key,
+            model="gpt-4.1",  # Fixed model for user simulator
+        )
+
+        try:
+            # Run the multi-agent conversation
+            conversation = await runner.run(task_obj, assistant_chat_client, user_simulator_chat_client)
+        except Exception:
+            # Handle failures gracefully - assign low reward to discourage problematic behavior
+            # Common issues: tool calling errors, timeout, invalid responses
+            traceback.print_exc()
+            return 0.0
+
+        # Use Tau2's built-in evaluation metrics
+        evaluation = runner.evaluate(task_obj, conversation, runner.termination_reason)
+
+        # Return the evaluation score
+        return evaluation  # noqa: RET504
+
+
+def main():
+    """Main entrypoint."""
+    # RL config with higher resource requirements and W&B logging
+    rl_training_config = {
+        "algorithm": {"adv_estimator": "grpo"},
+        "data": {
+            "train_batch_size": 8,
+            "max_prompt_length": 8192,
+            "max_response_length": 2048,
+        },
+        "actor_rollout_ref": {
+            "rollout": {
+                "tensor_model_parallel_size": 1,
+                "n": 8,  # Higher repetition for more data per task
+                "log_prob_micro_batch_size_per_gpu": 4,
+                "multi_turn": {"format": "hermes"},
+                "name": "vllm",
+                "gpu_memory_utilization": 0.8,  # Higher utilization for 80GB GPU
+            },
+            "actor": {
+                "ppo_mini_batch_size": 8,
+                "ppo_micro_batch_size_per_gpu": 4,
+                "optim": {"lr": 1e-6},
+                "use_kl_loss": False,
+                "clip_ratio_low": 0.2,
+                "clip_ratio_high": 0.3,
+                "fsdp_config": {
+                    "param_offload": True,
+                    "optimizer_offload": True,
+                },
+            },
+            # Reference model config
+            "ref": {
+                "log_prob_micro_batch_size_per_gpu": 8,
+                "fsdp_config": {"param_offload": True},
+            },
+            # Common configs for the model
+            "model": {
+                "path": "Qwen/Qwen2.5-1.5B-Instruct",
+                "use_remove_padding": True,
+                "enable_gradient_checkpointing": True,
+            },
+        },
+        "trainer": {
+            "n_gpus_per_node": 1,
+            "val_before_train": True,
+            "logger": ["console", "wandb"],  # Wandb for experiment tracking
+            "project_name": "agent-framework-lab-lightning",
+            "experiment_name": "tau2_agent",
+            "nnodes": 1,
+            "test_freq": 4,
+            "total_epochs": 8,
+        },
+    }
+
+    patch_env_set_state()  # Tau2-specific environment setup
+
+    train_dataset, val_dataset = _load_dataset()
+
+    # Key difference with math_agent: trained_agents parameter specifies which agents to train
+    # Only the assistant agent is trained; user simulator remains fixed
+    tau2_agent = Tau2Agent(trained_agents=ASSISTANT_AGENT_ID)
+
+    tracer = AgentFrameworkTracer()
+    trainer = Trainer(algorithm=VERL(rl_training_config), tracer=tracer, n_workers=4)
+    trainer.fit(tau2_agent, train_dataset, val_dataset=val_dataset)
+
+
+def debug():
+    """Debug mode for testing multi-agent setup and Tau2 integration."""
+    train_dataset, _ = _load_dataset()
+    tau2_agent = Tau2Agent(trained_agents=ASSISTANT_AGENT_ID)
+
+    openai_base_url = os.getenv("OPENAI_BASE_URL")
+    if openai_base_url is None:
+        raise ValueError("OPENAI_BASE_URL must be set")
+
+    patch_env_set_state()  # Required for Tau2 environment
+
+    # Test with resources dict (different from @rollout LLM parameter)
+    asyncio.run(
+        tau2_agent.rollout_async(
+            train_dataset[0],
+            resources={"main_llm": LLM(model="gpt-4.1", endpoint=openai_base_url)},
+            rollout=Rollout(rollout_id="dummy", input="dummy_input", start_time=time.time()),
+        )
+    )
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--debug", action="store_true")
+    args = parser.parse_args()
+    if args.debug:
+        debug()
+    else:
+        main()

python/packages/lab/lightning/tests/test_lightning.py

@@ -0,0 +1,163 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for lightning module."""
+
+# ruff: noqa
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+from agent_framework import AgentExecutor, AgentResponse, Agent, WorkflowBuilder, Workflow
+from agent_framework.openai import OpenAIChatCompletionClient
+from openai.types.chat import ChatCompletion, ChatCompletionMessage
+from openai.types.chat.chat_completion import Choice
+
+
+@pytest.fixture
+def workflow_two_agents():
+    """Test a workflow with two OpenAI chat agents where first agent's result passes to second agent."""
+
+    # Mock OpenAI responses
+    first_agent_response = ChatCompletion(
+        id="chatcmpl-123",
+        object="chat.completion",
+        created=1677652288,
+        model="gpt-4o",
+        choices=[
+            Choice(
+                index=0,
+                message=ChatCompletionMessage(role="assistant", content="Analyzed data shows trend upward"),
+                finish_reason="stop",
+            )
+        ],
+    )
+
+    second_agent_response = ChatCompletion(
+        id="chatcmpl-456",
+        object="chat.completion",
+        created=1677652289,
+        model="gpt-4o",
+        choices=[
+            Choice(
+                index=0,
+                message=ChatCompletionMessage(
+                    role="assistant",
+                    content="Based on the analysis 'Analyzed data shows trend upward', I recommend investing",
+                ),
+                finish_reason="stop",
+            )
+        ],
+    )
+
+    # Create mock OpenAI clients
+    with patch.dict(
+        "os.environ",
+        {
+            "OPENAI_API_KEY": "test-key",
+            "OPENAI_MODEL": "gpt-4o",
+        },
+    ):
+        first_chat_client = OpenAIChatCompletionClient()
+        second_chat_client = OpenAIChatCompletionClient()
+
+        # Mock the OpenAI API calls
+        with (
+            patch.object(
+                first_chat_client.client.chat.completions,
+                "create",
+                new_callable=AsyncMock,
+                return_value=first_agent_response,
+            ),
+            patch.object(
+                second_chat_client.client.chat.completions,
+                "create",
+                new_callable=AsyncMock,
+                return_value=second_agent_response,
+            ),
+        ):
+            # Create the two agents
+            analyzer_agent = Agent(
+                client=first_chat_client,
+                name="DataAnalyzer",
+                instructions="You are a data analyst. Analyze the given data and provide insights.",
+            )
+
+            advisor_agent = Agent(
+                client=second_chat_client,
+                name="InvestmentAdvisor",
+                instructions="You are an investment advisor. Based on analysis results, provide recommendations.",
+            )
+
+            analyzer_executor = AgentExecutor(id="analyzer", agent=analyzer_agent)
+            advisor_executor = AgentExecutor(id="advisor", agent=advisor_agent)
+
+            # Build workflow: analyzer -> advisor
+            workflow = (
+                WorkflowBuilder(start_executor=analyzer_executor).add_edge(analyzer_executor, advisor_executor).build()
+            )
+
+            yield workflow
+
+
+async def test_openai_workflow_two_agents(workflow_two_agents: Workflow):
+    events = await workflow_two_agents.run("Please analyze the quarterly sales data")
+
+    # Get all output events with AgentResponse
+    agent_outputs = [event.data for event in events if event.type == "output" and isinstance(event.data, AgentResponse)]
+
+    # Check that we have outputs from both agents
+    assert len(agent_outputs) == 2
+    assert any("Analyzed data shows trend upward" in str(output) for output in agent_outputs)
+    assert any(
+        "Based on the analysis 'Analyzed data shows trend upward', I recommend investing" in str(output)
+        for output in agent_outputs
+    )
+
+
+@pytest.mark.resource_intensive
+async def test_observability(workflow_two_agents: Workflow):
+    r"""Expected trace tree:
+
+                    [workflow.run]
+                    /      \
+            [analyzer]      [advisor]
+            /      \          /    \
+    [DataAnalyzer] [send] [Investment] [send]
+            |                    |
+        [chat gpt-4o]        [chat gpt-4o]
+    """
+    pytest.importorskip("agentlightning")
+    from agent_framework_lab_lightning import AgentFrameworkTracer
+    from agentlightning.adapter import TracerTraceToTriplet
+
+    tracer = AgentFrameworkTracer()
+    try:
+        tracer.init()
+        tracer.init_worker(0)
+
+        async with tracer.trace_context():
+            await workflow_two_agents.run("Please analyze the quarterly sales data")
+
+        triplets = TracerTraceToTriplet(agent_match=None, llm_call_match="chat").adapt(tracer.get_last_trace())
+        assert len(triplets) == 2
+
+        triplets = TracerTraceToTriplet(agent_match="analyzer", llm_call_match="chat").adapt(tracer.get_last_trace())
+        assert len(triplets) == 1
+
+        triplets = TracerTraceToTriplet(agent_match="advisor", llm_call_match="chat").adapt(tracer.get_last_trace())
+        assert len(triplets) == 1
+
+        # Parent agent is not matched
+        triplets = TracerTraceToTriplet(agent_match="DataAnalyzer", llm_call_match="chat").adapt(
+            tracer.get_last_trace()
+        )
+        assert len(triplets) == 0
+
+        triplets = TracerTraceToTriplet(agent_match="InvestmentAdvisor|advisor", llm_call_match="chat").adapt(
+            tracer.get_last_trace()
+        )
+        assert len(triplets) == 1
+
+    finally:
+        tracer.teardown_worker(0)
+        tracer.teardown()

python/packages/lab/namespace/agent_framework/__init__.py

@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)

python/packages/lab/namespace/agent_framework/lab/__init__.py

@@ -0,0 +1,4 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# This makes agent_framework.lab a namespace package
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)

python/packages/lab/namespace/agent_framework/lab/gaia/__init__.py

@@ -0,0 +1,28 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_gaia import (
+    GAIA,
+    Evaluation,
+    Evaluator,
+    GAIATelemetryConfig,
+    Prediction,
+    Task,
+    TaskResult,
+    TaskRunner,
+    gaia_scorer,
+    viewer_main,
+)
+
+__all__ = [
+    "GAIA",
+    "Evaluation",
+    "Evaluator",
+    "GAIATelemetryConfig",
+    "Prediction",
+    "Task",
+    "TaskResult",
+    "TaskRunner",
+    "gaia_scorer",
+    "viewer_main",
+]

python/packages/lab/namespace/agent_framework/lab/lightning/__init__.py

@@ -0,0 +1,6 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_lightning import AgentFrameworkTracer
+
+__all__ = ["AgentFrameworkTracer"]

python/packages/lab/namespace/agent_framework/lab/tau2/__init__.py

@@ -0,0 +1,20 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+# Import and re-export from the actual implementation
+from agent_framework_lab_tau2 import (
+    ASSISTANT_AGENT_ID,
+    ORCHESTRATOR_ID,
+    USER_SIMULATOR_ID,
+    TaskRunner,
+    patch_env_set_state,
+    unpatch_env_set_state,
+)
+
+__all__ = [
+    "ASSISTANT_AGENT_ID",
+    "ORCHESTRATOR_ID",
+    "USER_SIMULATOR_ID",
+    "TaskRunner",
+    "patch_env_set_state",
+    "unpatch_env_set_state",
+]

python/packages/lab/pyproject.toml

@@ -0,0 +1,171 @@
+[project]
+name = "agent-framework-lab"
+description = "Experimental modules for Microsoft Agent Framework"
+authors = [{ name = "Microsoft", email = "af-support@microsoft.com"}]
+readme = "README.md"
+requires-python = ">=3.10"
+version = "1.0.0b260709"
+license-files = ["LICENSE"]
+urls.homepage = "https://aka.ms/agent-framework"
+urls.source = "https://github.com/microsoft/agent-framework/tree/main/python"
+urls.release_notes = "https://github.com/microsoft/agent-framework/releases?q=tag%3Apython-1&expanded=true"
+urls.issues = "https://github.com/microsoft/agent-framework/issues"
+classifiers = [
+  "License :: OSI Approved :: MIT License",
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+]
+dependencies = [
+  "agent-framework-core>=1.11.0,<2",
+]
+
+[project.optional-dependencies]
+# GAIA benchmark module dependencies
+gaia = [
+  "pydantic>=2.0.0",
+  "opentelemetry-api>=1.39.0",
+  "opentelemetry-sdk>=1.39.0,<2",
+  "tqdm>=4.60.0",
+  "huggingface-hub>=0.20.0",
+  "orjson>=3.10.7,<4",
+  "pyarrow>=18.0.0",  # For reading parquet files
+]
+
+# Lightning RL training module dependencies
+lightning = [
+  "agentlightning>=0.2.0,<0.4.0",
+]
+
+# TAU2 benchmark module dependencies
+tau2 = [
+  "pydantic>=2.0.0",
+  "tiktoken>=0.11.0",
+  "loguru>=0.7.3",
+  "numpy",
+]
+
+# Dependencies for math-related training
+math = [
+  "sympy>=1.13.0",
+]
+
+[dependency-groups]
+dev = [
+    "uv==0.11.28",
+    "ruff==0.15.20",
+    "pytest==9.1.1",
+    "mypy==2.2.0",
+    "pyright==1.1.411",
+    #tasks
+    "poethepoet==0.48.0",
+    "rich>=13.7.1,<15.0.0",
+    "tomli==2.4.1",
+    "tomli-w==1.2.0",
+    "prek==0.4.8",
+]
+# tau2 is an executable optional feature fetched from source because it is not available on PyPI.
+tau2 = [
+    "tau2@ git+https://github.com/sierra-research/tau2-bench@5ba9e3e56db57c5e4114bf7f901291f09b2c5619",
+]
+
+[project.scripts]
+gaia_viewer = "agent_framework_lab_gaia:viewer_main"
+lightning = "agent_framework_lab_lightning:main"
+
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = [
+  "agent_framework_lab_gaia",
+  "agent_framework_lab_lightning",
+  "agent_framework_lab_tau2",
+  "agent_framework.lab.gaia",
+  "agent_framework.lab.lightning",
+  "agent_framework.lab.tau2",
+]
+
+[tool.setuptools.package-dir]
+"agent_framework_lab_gaia" = "gaia/agent_framework_lab_gaia"
+"agent_framework_lab_lightning" = "lightning/agent_framework_lab_lightning"
+"agent_framework_lab_tau2" = "tau2/agent_framework_lab_tau2"
+"agent_framework.lab.gaia" = "namespace/agent_framework/lab/gaia"
+"agent_framework.lab.lightning" = "namespace/agent_framework/lab/lightning"
+"agent_framework.lab.tau2" = "namespace/agent_framework/lab/tau2"
+
+[tool.setuptools.package-data]
+agent_framework_lab_gaia = ["py.typed"]
+agent_framework_lab_lightning = ["py.typed"]
+agent_framework_lab_tau2 = ["py.typed"]
+
+[tool.ruff]
+extend = "../../pyproject.toml"
+extend-exclude = ["**/data/**"]
+
+[tool.ruff.lint]
+ignore = [
+    "T201",  # Allow print statements in experimental/lab code for debugging purposes.
+    "ASYNC230",  # Allow 'await' outside of async functions in test and experimental code.
+    "INP001",  # Ignore missing __init__.py in namespace packages.
+    "RUF029",  # Allow use of 'assert' statements; assertions are used for internal checks in experimental code.
+    "ASYNC240",  # Allow 'async for' outside of async functions in test and experimental code.
+]
+
+[tool.coverage.run]
+omit = [
+    "**/__init__.py"
+]
+
+[tool.pyright]
+extends = "../../pyproject.toml"
+include = ["gaia/agent_framework_lab_gaia", "lightning/agent_framework_lab_lightning", "tau2/agent_framework_lab_tau2"]
+exclude = ['gaia/tests', 'lightning/tests', 'tau2/tests', 'namespace', '**/samples']
+
+[tool.bandit]
+targets = ["agent_framework_lab_gaia", "agent_framework_lab_lightning", "agent_framework_lab_tau2"]
+exclude_dirs = ["gaia/tests", "lightning/tests", "tau2/tests"]
+
+[tool.poe]
+include = "../../shared_tasks.toml"
+
+[tool.poe.tasks.test]
+help = "Run the default lab unit test suite."
+cmd = 'pytest -m "not integration and not resource_intensive" --cov-report=term-missing:skip-covered --junitxml=test-results.xml'
+
+[tool.poe.tasks.test-gaia]
+help = "Run the GAIA lab test suite."
+cmd = "pytest gaia/tests --cov=agent_framework_lab_gaia --cov-report=term-missing:skip-covered"
+
+[tool.poe.tasks.test-lightning]
+help = "Run the Lightning lab test suite."
+cmd = "pytest lightning/tests --cov=agent_framework_lab_lightning --cov-report=term-missing:skip-covered"
+
+[tool.poe.tasks.test-tau2]
+help = "Run the Tau2 lab test suite."
+cmd = "pytest tau2/tests --cov=agent_framework_lab_tau2 --cov-report=term-missing:skip-covered"
+
+[tool.poe.tasks.build]
+help = "Skip build for the lab package."
+cmd = "echo 'Skipping build'"
+
+[tool.poe.tasks.publish]
+help = "Skip publish for the lab package."
+cmd = "echo 'Skipping publish'"
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+addopts = "--strict-markers --strict-config"
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+markers = [
+  "unit: marks tests as unit tests",
+  "integration: marks tests as integration tests",
+  "resource_intensive: marks tests that are expensive and excluded from default package test runs",
+]

python/packages/lab/tau2/README.md

@@ -0,0 +1,198 @@
+# Agent Framework Lab - τ²-bench
+
+τ²-bench implements a simulation framework for evaluating customer service agents across various domains.
+
+> **Note**: This module is part of the consolidated `agent-framework-lab` package. Install the package with the `tau2` extra to use this module.
+
+The framework orchestrates conversations between two AI agents:
+
+- **Customer Service Agent**: Follows domain-specific policies and has access to tools (e.g., booking systems, databases)
+- **User Simulator**: Simulates realistic customer behavior with specific goals and scenarios
+
+Each evaluation runs a multi-turn conversation where the user simulator presents a customer service scenario, and the agent must resolve it following the domain policy while using available tools appropriately. The results are evaluated using τ²'s comprehensive evaluation system.
+
+## Supported Domains
+
+| Domain      | Status            | Description                                                |
+| ----------- | ----------------- | ---------------------------------------------------------- |
+| **airline** | ✅ Supported      | Customer service for airline booking, changes, and support |
+| **retail**  | 🚧 In Development | E-commerce customer support scenarios                      |
+| **telecom** | 🚧 In Development | Telecommunications service support                         |
+
+_Note: Currently only the airline domain is fully supported._
+
+## Installation
+
+Install the agent-framework-lab package with TAU2 dependencies:
+
+```bash
+pip install "agent-framework-lab[tau2]"
+```
+
+**Important:** You must also install the tau2-bench package from source:
+
+```bash
+pip install "tau2 @ git+https://github.com/sierra-research/tau2-bench@5ba9e3e56db57c5e4114bf7f901291f09b2c5619"
+```
+
+Download data from [Tau2-Bench](https://github.com/sierra-research/tau2-bench):
+
+```bash
+git clone https://github.com/sierra-research/tau2-bench.git
+mv tau2-bench/data/ .
+rm -rf tau2-bench
+```
+
+Export the data directory to `TAU2_DATA_DIR` environment variable:
+
+```bash
+export TAU2_DATA_DIR="data"
+```
+
+## Quick Start
+
+### Running a Single Task
+
+```python
+import asyncio
+from agent_framework.openai import OpenAIChatClient
+from agent_framework.lab.tau2 import TaskRunner
+from tau2.domains.airline.environment import get_tasks
+
+async def run_single_task():
+    # Initialize the task runner
+    runner = TaskRunner(max_steps=50)
+
+    # Set up your LLM clients
+    assistant_client = OpenAIChatClient(
+        base_url="https://api.openai.com/v1",
+        api_key="your-api-key",
+        model="gpt-4o"
+    )
+    user_client = OpenAIChatClient(
+        base_url="https://api.openai.com/v1",
+        api_key="your-api-key",
+        model="gpt-4o-mini"
+    )
+
+    # Get a task and run it
+    tasks = get_tasks()
+    task = tasks[0]  # Run the first task
+
+    conversation = await runner.run(task, assistant_client, user_client)
+    reward = runner.evaluate(task, conversation, runner.termination_reason)
+
+    print(f"Task completed with reward: {reward}")
+
+# Run the example
+asyncio.run(run_single_task())
+```
+
+### Running the Full Benchmark
+
+Use the provided script to run the complete benchmark:
+
+```bash
+# Run with default models (gpt-4.1 for both agent and user)
+python samples/run_benchmark.py
+
+# Use custom models
+python samples/run_benchmark.py --assistant gpt-4o --user gpt-4o-mini
+
+# Debug a specific task
+python samples/run_benchmark.py --debug-task-id task_001 --assistant gpt-4o
+
+# Limit conversation length
+python samples/run_benchmark.py --max-steps 20
+```
+
+## Results (on Airline Domain)
+
+The following results are reproduced from our implementation of τ²-bench with `samples/run_benchmark.py`. It shows the average success rate over the dataset of 50 tasks.
+
+| Agent Model  | User Model  | Success Rate |
+| ------------ | ----------- | ------------ |
+| gpt-5        | gpt-4.1     | 62.0%        |
+| gpt-5-mini   | gpt-4.1     | 52.0%        |
+| gpt-4.1      | gpt-4.1     | 60.0%        |
+| gpt-4.1-mini | gpt-4.1     | 50.0%        |
+| gpt-4.1      | gpt-4o-mini | 42.0%        |
+| gpt-4o       | gpt-4.1     | 42.0%        |
+| gpt-4o-mini  | gpt-4.1     | 26.0%        |
+
+## Advanced Usage
+
+### Environment Configuration
+
+Set required environment variables:
+
+```bash
+export OPENAI_BASE_URL="https://api.openai.com/v1"
+export OPENAI_API_KEY="your-api-key"
+
+# Optional: for custom endpoints
+export OPENAI_BASE_URL="https://your-custom-endpoint.com/v1"
+```
+
+### Custom Agent Implementation
+
+```python
+from agent_framework.lab.tau2 import TaskRunner
+from agent_framework import Agent
+
+class CustomTaskRunner(TaskRunner):
+    def assistant_agent(self, assistant_chat_client):
+        # Override to customize the assistant agent
+        return Agent(
+            client=assistant_chat_client,
+            instructions="Your custom system prompt here",
+            # Add custom tools, temperature, etc.
+        )
+
+    def user_simulator(self, user_chat_client, task):
+        # Override to customize the user simulator
+        return Agent(
+            client=user_chat_client,
+            instructions="Custom user simulator prompt",
+        )
+```
+
+### Custom Workflow Integration
+
+```python
+from agent_framework import WorkflowBuilder, AgentExecutor
+from agent_framework.lab.tau2 import TaskRunner
+
+class WorkflowTaskRunner(TaskRunner):
+    def build_conversation_workflow(self, assistant_agent, user_simulator_agent):
+        # Create agent executors
+        assistant_executor = AgentExecutor(assistant_agent, id="assistant_agent")
+        user_executor = AgentExecutor(user_simulator_agent, id="user_simulator")
+
+        # Build a custom workflow with start executor
+        builder = WorkflowBuilder(start_executor=assistant_executor)
+        builder.add_edge(assistant_executor, user_executor)
+        builder.add_edge(user_executor, assistant_executor, condition=self.should_not_stop)
+
+        return builder.build()
+```
+
+### Utility Functions
+
+```python
+from agent_framework.lab.tau2 import patch_env_set_state, unpatch_env_set_state
+
+# Enable compatibility patches for τ²-bench integration
+patch_env_set_state()
+
+# Disable patches when done
+unpatch_env_set_state()
+```
+
+## Contributing
+
+This package is part of the Microsoft Agent Framework Lab. Please see the main repository for contribution guidelines.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

python/packages/lab/tau2/agent_framework_lab_tau2/__init__.py

@@ -0,0 +1,22 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tau2 Benchmark for Agent Framework."""
+
+import importlib.metadata
+
+from ._tau2_utils import patch_env_set_state, unpatch_env_set_state
+from .runner import ASSISTANT_AGENT_ID, ORCHESTRATOR_ID, USER_SIMULATOR_ID, TaskRunner
+
+try:
+    __version__ = importlib.metadata.version(__name__)
+except importlib.metadata.PackageNotFoundError:
+    __version__ = "0.0.0"  # Fallback for development mode
+
+__all__ = [
+    "ASSISTANT_AGENT_ID",
+    "ORCHESTRATOR_ID",
+    "USER_SIMULATOR_ID",
+    "TaskRunner",
+    "patch_env_set_state",
+    "unpatch_env_set_state",
+]

python/packages/lab/tau2/agent_framework_lab_tau2/_message_utils.py

@@ -0,0 +1,121 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+from typing import Any
+
+from agent_framework._types import Content, Message
+from loguru import logger
+
+
+def _get_role_value(role: Any) -> str:
+    """Get the string value of a role, handling both enum and string."""
+    return role.value if hasattr(role, "value") else str(role)
+
+
+def flip_messages(messages: list[Message]) -> list[Message]:
+    """Flip message roles between assistant and user for role-playing scenarios.
+
+    Used in agent simulations where the assistant's messages become user inputs
+    and vice versa. Function calls are filtered out when flipping assistant
+    messages to user messages (since users typically don't make function calls).
+    """
+
+    def filter_out_function_calls(messages: list[Content]) -> list[Content]:
+        """Remove function call content from message contents."""
+        return [content for content in messages if content.type != "function_call"]
+
+    flipped_messages: list[Message] = []
+    for msg in messages:
+        role_value = _get_role_value(msg.role)
+        if role_value == "assistant":
+            # Flip assistant to user
+            contents = filter_out_function_calls(msg.contents)
+            if contents:
+                flipped_msg = Message(
+                    role="user",
+                    # The function calls will cause 400 when role is user
+                    contents=contents,
+                    author_name=msg.author_name,
+                    message_id=msg.message_id,
+                )
+                flipped_messages.append(flipped_msg)
+        elif role_value == "user":
+            # Flip user to assistant
+            flipped_msg = Message(
+                role="assistant", contents=msg.contents, author_name=msg.author_name, message_id=msg.message_id
+            )
+            flipped_messages.append(flipped_msg)
+        elif role_value == "tool":
+            # Skip tool messages
+            pass
+        else:
+            # Keep other roles as-is (system, tool, etc.)
+            flipped_messages.append(msg)
+    return flipped_messages
+
+
+def log_messages(messages: list[Message]) -> None:
+    """Log messages with colored output based on role and content type.
+
+    Provides visual debugging by color-coding different message roles and
+    content types. Escapes HTML-like characters to prevent log formatting issues.
+    """
+    logger_ = logger.opt(colors=True)
+    for msg in messages:
+        role_value = _get_role_value(msg.role)
+        # Handle different content types
+        if hasattr(msg, "contents") and msg.contents:
+            for content in msg.contents:
+                if hasattr(content, "type"):
+                    if content.type == "text":
+                        escape_text = content.text.replace("<", r"\<")  # type: ignore[union-attr]
+                        if role_value == "system":
+                            logger_.info(f"<cyan>[SYSTEM]</cyan> {escape_text}")
+                        elif role_value == "user":
+                            logger_.info(f"<green>[USER]</green> {escape_text}")
+                        elif role_value == "assistant":
+                            logger_.info(f"<blue>[ASSISTANT]</blue> {escape_text}")
+                        elif role_value == "tool":
+                            logger_.info(f"<yellow>[TOOL]</yellow> {escape_text}")
+                        else:
+                            logger_.info(f"<magenta>[{role_value.upper()}]</magenta> {escape_text}")
+                    elif content.type == "function_call":
+                        function_call_text = f"{content.name}({content.arguments})"
+                        function_call_text = function_call_text.replace("<", r"\<")
+                        logger_.info(f"<yellow>[TOOL_CALL]</yellow> 🔧 {function_call_text}")
+                    elif content.type == "function_result":
+                        function_result_text = f"ID:{content.call_id} -> {content.result}"
+                        function_result_text = function_result_text.replace("<", r"\<")
+                        logger_.info(f"<yellow>[TOOL_RESULT]</yellow> 🔨 {function_result_text}")
+                    else:
+                        content_text = str(content).replace("<", r"\<")
+                        logger_.info(f"<magenta>[{role_value.upper()}] ({content.type})</magenta> {content_text}")
+                else:
+                    # Fallback for content without type
+                    text_content = str(content).replace("<", r"\<")
+                    if role_value == "system":
+                        logger_.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
+                    elif role_value == "user":
+                        logger_.info(f"<green>[USER]</green> {text_content}")
+                    elif role_value == "assistant":
+                        logger_.info(f"<blue>[ASSISTANT]</blue> {text_content}")
+                    elif role_value == "tool":
+                        logger_.info(f"<yellow>[TOOL]</yellow> {text_content}")
+                    else:
+                        logger_.info(f"<magenta>[{role_value.upper()}]</magenta> {text_content}")
+        elif hasattr(msg, "text") and msg.text:
+            # Handle simple text messages
+            text_content = msg.text.replace("<", r"\<")
+            if role_value == "system":
+                logger_.info(f"<cyan>[SYSTEM]</cyan> {text_content}")
+            elif role_value == "user":
+                logger_.info(f"<green>[USER]</green> {text_content}")
+            elif role_value == "assistant":
+                logger_.info(f"<blue>[ASSISTANT]</blue> {text_content}")
+            elif role_value == "tool":
+                logger_.info(f"<yellow>[TOOL]</yellow> {text_content}")
+            else:
+                logger_.info(f"<magenta>[{role_value.upper()}]</magenta> {text_content}")
+        else:
+            # Fallback for other message formats
+            text_content = str(msg).replace("<", r"\<")
+            logger_.info(f"<magenta>[{role_value.upper()}]</magenta> {text_content}")

python/packages/lab/tau2/agent_framework_lab_tau2/_sliding_window.py

@@ -0,0 +1,121 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import json
+from typing import Any
+
+import tiktoken
+from agent_framework import InMemoryHistoryProvider, Message
+from loguru import logger
+
+
+class SlidingWindowHistoryProvider(InMemoryHistoryProvider):
+    """A token-aware sliding window implementation of InMemoryHistoryProvider.
+
+    Stores all messages in session state but returns a truncated window from
+    ``get_messages`` that fits within ``max_tokens``. Automatically removes
+    oldest messages and leading tool messages to ensure valid conversation flow.
+    """
+
+    def __init__(
+        self,
+        source_id: str = InMemoryHistoryProvider.DEFAULT_SOURCE_ID,
+        *,
+        max_tokens: int = 3800,
+        system_message: str | None = None,
+        tool_definitions: Any | None = None,
+    ):
+        super().__init__(source_id)
+        self.max_tokens = max_tokens
+        self.system_message = system_message  # Included in token count
+        self.tool_definitions = tool_definitions
+        # An estimation based on a commonly used vocab table
+        self.encoding = tiktoken.get_encoding("o200k_base")
+
+    async def get_messages(
+        self, session_id: str | None, *, state: dict[str, Any] | None = None, **kwargs: Any
+    ) -> list[Message]:
+        """Retrieve messages from session state, truncated to fit within max_tokens."""
+        all_messages = await super().get_messages(session_id, state=state, **kwargs)
+        return self._truncate(list(all_messages))
+
+    def _truncate(self, messages: list[Message]) -> list[Message]:
+        """Truncate messages to fit within max_tokens and remove leading tool messages."""
+        while len(messages) > 0 and self._get_token_count(messages) > self.max_tokens:
+            logger.warning("Messages exceed max tokens. Truncating oldest message.")
+            messages.pop(0)
+        # Remove leading tool messages
+        while len(messages) > 0:
+            if messages[0].role != "tool":
+                break
+            logger.warning("Removing leading tool message because tool result cannot be the first message.")
+            messages.pop(0)
+        return messages
+
+    def _get_token_count(self, messages: list[Message]) -> int:
+        """Estimate token count for a list of messages using tiktoken.
+
+        Returns:
+            Estimated token count
+        """
+        total_tokens = 0
+
+        # Add system message tokens if provided
+        if self.system_message:
+            total_tokens += len(self.encoding.encode(self.system_message))
+            total_tokens += 4  # Extra tokens for system message formatting
+
+        for msg in messages:
+            # Add 4 tokens per message for role, formatting, etc.
+            total_tokens += 4
+
+            # Handle different content types
+            if hasattr(msg, "contents") and msg.contents:
+                for content in msg.contents:
+                    if hasattr(content, "type"):
+                        if content.type == "text":
+                            total_tokens += len(self.encoding.encode(content.text))  # type: ignore[arg-type]
+                        elif content.type == "function_call":
+                            total_tokens += 4
+                            # Serialize function call and count tokens
+                            func_call_data = {
+                                "name": content.name,
+                                "arguments": content.arguments,
+                            }
+                            total_tokens += self._estimate_any_object_token_count(func_call_data)
+                        elif content.type == "function_result":
+                            total_tokens += 4
+                            # Serialize function result and count tokens
+                            func_result_data = {
+                                "call_id": content.call_id,
+                                "result": content.result,
+                            }
+                            total_tokens += self._estimate_any_object_token_count(func_result_data)
+                        else:
+                            # For other content types, serialize the whole content
+                            total_tokens += self._estimate_any_object_token_count(content)
+                    else:
+                        # Content without type, treat as text
+                        total_tokens += self._estimate_any_object_token_count(content)
+            elif hasattr(msg, "text") and msg.text:
+                # Simple text message
+                total_tokens += self._estimate_any_object_token_count(msg.text)
+
+        if total_tokens > self.max_tokens / 2:
+            logger.opt(colors=True).warning(
+                f"<yellow>Total tokens {total_tokens} is "
+                f"{total_tokens / self.max_tokens * 100:.0f}% "
+                f"of max tokens {self.max_tokens}</yellow>"
+            )
+        elif total_tokens > self.max_tokens:
+            logger.opt(colors=True).warning(
+                f"<red>Total tokens {total_tokens} is over max tokens {self.max_tokens}. Will truncate messages.</red>"
+            )
+
+        return total_tokens
+
+    def _estimate_any_object_token_count(self, obj: Any) -> int:
+        try:
+            serialized = json.dumps(obj)
+        except Exception:
+            serialized = str(obj)
+        return len(self.encoding.encode(serialized))

python/packages/lab/tau2/agent_framework_lab_tau2/_tau2_utils.py

@@ -0,0 +1,257 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import json
+from collections.abc import Mapping
+from copy import deepcopy
+from typing import Any, TypeGuard, cast
+
+import numpy as np
+from agent_framework._tools import FunctionTool
+from agent_framework._types import Message
+from loguru import logger
+from pydantic import BaseModel
+from tau2.data_model.message import (
+    AssistantMessage,
+    SystemMessage,
+    ToolCall,
+    ToolMessage,
+    UserMessage,
+)
+from tau2.data_model.message import (
+    Message as Tau2Message,
+)
+from tau2.data_model.tasks import EnvFunctionCall, InitializationData
+from tau2.environment.environment import Environment
+from tau2.environment.tool import Tool
+
+_original_set_state = Environment.set_state
+
+
+def _to_str(value: object, default: str = "") -> str:
+    if isinstance(value, str):
+        return value
+    if value is None:
+        return default
+    return str(value)
+
+
+def _is_any_list(value: Any) -> TypeGuard[list[Any]]:
+    return isinstance(value, list)
+
+
+def _is_any_mapping(value: Any) -> TypeGuard[Mapping[Any, Any]]:
+    return isinstance(value, Mapping)
+
+
+def _is_any_sequence(value: Any) -> TypeGuard[list[Any] | tuple[Any, ...] | set[Any]]:
+    return isinstance(value, (list, tuple, set))
+
+
+def convert_tau2_tool_to_function_tool(tau2_tool: Tool) -> FunctionTool:
+    """Convert a tau2 Tool to a FunctionTool for agent framework compatibility.
+
+    Creates a wrapper that preserves the tool's interface while ensuring
+    results are deep-copied to prevent unintended mutations.
+    """
+
+    def wrapped_func(**kwargs: Any) -> Any:
+        result = tau2_tool(**kwargs)
+        # Deep copy to prevent mutations of returned data
+        return result.model_copy(deep=True) if isinstance(result, BaseModel) else deepcopy(result)
+
+    return FunctionTool(
+        name=tau2_tool.name,
+        description=tau2_tool._get_description(),  # pyright: ignore[reportPrivateUsage]
+        func=wrapped_func,
+        input_model=tau2_tool.params,
+    )
+
+
+def convert_agent_framework_messages_to_tau2_messages(messages: list[Message]) -> list[Tau2Message]:
+    """Convert agent framework ChatMessages to tau2 Message objects.
+
+    Handles role mapping, text extraction, function calls, and function results.
+    Function results are converted to separate ToolMessage instances.
+    """
+    tau2_messages: list[Tau2Message] = []
+
+    for msg in messages:
+        role_str = str(msg.role)
+
+        # Extract text content from all text-type contents
+        text_contents = [c for c in msg.contents if hasattr(c, "text") and hasattr(c, "type") and c.type == "text"]
+        content_parts: list[str] = [_to_str(getattr(c, "text", "")) for c in text_contents]
+        content_value = " ".join(content_parts)
+
+        # Extract function calls and convert to ToolCall objects
+        function_calls = [c for c in msg.contents if hasattr(c, "type") and c.type == "function_call"]
+        tool_calls: list[ToolCall] | None = None
+        if function_calls:
+            tool_calls = []
+            for fc in function_calls:
+                arguments = fc.parse_arguments() or {}
+                tool_call = ToolCall(
+                    id=_to_str(fc.call_id),
+                    name=_to_str(fc.name),
+                    arguments=arguments,
+                    requestor="assistant" if role_str == "assistant" else "user",
+                )
+                tool_calls.append(tool_call)
+
+        # Extract function results for separate ToolMessage creation
+        function_results = [c for c in msg.contents if hasattr(c, "type") and c.type == "function_result"]
+
+        # Create main message based on role
+        if role_str == "system":
+            tau2_messages.append(SystemMessage(role="system", content=content_value))
+        elif role_str == "user":
+            tau2_messages.append(UserMessage(role="user", content=content_value, tool_calls=tool_calls))
+        elif role_str == "assistant":
+            tau2_messages.append(AssistantMessage(role="assistant", content=content_value, tool_calls=tool_calls))
+        elif role_str == "tool":
+            # Tool messages are handled as function results below
+            pass
+
+        # Convert function results to separate ToolMessage instances
+        for fr in function_results:
+            dumpable_content = _dump_function_result(fr.result)
+            content = dumpable_content if isinstance(dumpable_content, str) else json.dumps(dumpable_content)
+            tool_msg = ToolMessage(
+                id=_to_str(fr.call_id),
+                role="tool",
+                content=content,
+                requestor="assistant",  # Most tool calls originate from assistant
+                error=fr.exception is not None,
+            )
+            tau2_messages.append(tool_msg)
+
+    return tau2_messages
+
+
+def patch_env_set_state() -> None:
+    """Patch Environment.set_state to allow inconsistent tool call results.
+
+    Modifies the original method to log warnings instead of raising errors
+    when actual tool results differ from expected results, enabling more
+    flexible testing and development workflows.
+    """
+
+    def set_state(
+        self: Any,
+        initialization_data: InitializationData | None,
+        initialization_actions: list[EnvFunctionCall] | None,
+        message_history: list[Tau2Message],
+    ) -> None:
+        if self.solo_mode and any(isinstance(message, UserMessage) for message in message_history):
+            raise ValueError("User messages are not allowed in solo mode")
+
+        def get_actions_from_messages(messages: list[Tau2Message]) -> list[tuple[ToolCall, ToolMessage]]:
+            """Get the actions from the messages."""
+            messages = deepcopy(messages)[::-1]
+            actions: list[tuple[ToolCall, ToolMessage]] = []
+            while messages:
+                message = messages.pop()
+                if isinstance(message, ToolMessage):
+                    raise ValueError("Tool message not expected. Tool messages should always follow a tool call.")
+                if isinstance(message, (AssistantMessage, UserMessage)) and message.is_tool_call():
+                    tool_calls = message.tool_calls
+                    if tool_calls is None:
+                        raise ValueError("Tool message expected. Got None.")
+                    for tc in tool_calls:
+                        if len(messages) == 0:
+                            raise ValueError("Tool message expected. Got None.")
+                        tm = messages.pop()
+                        if not isinstance(tm, ToolMessage):
+                            raise ValueError(f"Tool message expected. Got {type(tm)}")
+                        if tc.id != tm.id:
+                            raise ValueError(f"Tool call id mismatch. Got {tc.id} and {tm.id}")
+                        actions.append((tc, tm))
+
+            return actions
+
+        if initialization_data is not None:
+            agent_data = cast(object, getattr(initialization_data, "agent_data", None))
+            if isinstance(agent_data, dict):
+                self.tools.update_db(cast(dict[str, Any], agent_data))
+
+            user_data = cast(object, getattr(initialization_data, "user_data", None))
+            if isinstance(user_data, dict):
+                self.user_tools.update_db(cast(dict[str, Any], user_data))
+
+        if initialization_actions is not None:
+            for action in initialization_actions:
+                self.run_env_function_call(action)
+
+        action_responses = get_actions_from_messages(message_history)
+        for tool_call, expected_response in action_responses:
+            response = self.get_response(tool_call)
+            content = _recursive_json_deserialize(response.content)
+            expected_content = _recursive_json_deserialize(expected_response.content)
+            if content != expected_content:
+                diff = f"Tool call:\n{tool_call}\n\nReturned:\n{response}\n\nExpected:\n{expected_response}"
+                if isinstance(content, str) and content.startswith("Error:"):
+                    # If the tool call resulted in an error, the difference can be ignored
+                    logger.warning(f"Tool call resulted in an error. Ignoring the difference.\n{diff}")
+                else:
+                    raise ValueError(
+                        f"Tool call:\n{tool_call}\n\nReturned:\n{response}\n\nExpected:\n{expected_response}"
+                    )
+        self.sync_tools()
+
+    Environment.set_state = set_state
+
+
+def unpatch_env_set_state() -> None:
+    Environment.set_state = _original_set_state
+
+
+def _dump_function_result(result: Any) -> Any:
+    if isinstance(result, BaseModel):
+        return result.model_dump_json()
+    if _is_any_list(result):
+        return [_dump_function_result(item) for item in result]
+    if isinstance(result, dict):
+        result_dict = cast(dict[str, Any], result)
+        return {k: _dump_function_result(v) for k, v in result_dict.items()}
+    if result is None:
+        return None
+    return result
+
+
+def _to_native(obj: Any) -> Any:
+    """Convert data retrieved from Panquet to data usable in AGL server."""
+    # 1) Arrays -> list (then recurse)
+    if isinstance(obj, np.ndarray):
+        return _to_native(obj.tolist())
+
+    # 2) NumPy scalar types -> Python scalars
+    if isinstance(obj, np.generic):
+        return _to_native(obj.item())
+
+    # 3) Dict-like -> dict
+    if _is_any_mapping(obj):
+        return {_to_native(k): _to_native(v) for k, v in obj.items()}
+
+    # 4) Lists/Tuples/Sets -> list
+    if _is_any_sequence(obj):
+        return [_to_native(x) for x in obj]
+
+    # 5) Anything else: leave as-is
+    return obj
+
+
+def _recursive_json_deserialize(obj: Any) -> Any:
+    """Recursively deserialize a JSON object."""
+    if isinstance(obj, str):
+        try:
+            deserialized = json.loads(obj)
+            return _recursive_json_deserialize(deserialized)
+        except (json.JSONDecodeError, TypeError):
+            return obj
+    elif _is_any_list(obj):
+        return [_recursive_json_deserialize(item) for item in obj]
+    elif isinstance(obj, dict):
+        typed_obj = cast(dict[str, Any], obj)
+        return {k: _recursive_json_deserialize(v) for k, v in typed_obj.items()}
+    else:
+        return obj

python/packages/lab/tau2/agent_framework_lab_tau2/runner.py

@@ -0,0 +1,440 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+from __future__ import annotations
+
+import uuid
+from typing import Any, cast
+
+from agent_framework import (
+    Agent,
+    AgentExecutor,
+    AgentExecutorRequest,
+    AgentExecutorResponse,
+    AgentResponse,
+    FunctionExecutor,
+    InMemoryHistoryProvider,
+    Message,
+    SupportsChatGetResponse,
+    Workflow,
+    WorkflowBuilder,
+    WorkflowContext,
+)
+from loguru import logger
+from tau2.data_model.simulation import SimulationRun, TerminationReason
+from tau2.data_model.tasks import Task
+from tau2.domains.airline.environment import get_environment
+from tau2.evaluator.evaluator import EvaluationType, RewardInfo, evaluate_simulation
+from tau2.user.user_simulator import (
+    OUT_OF_SCOPE,
+    STOP,
+    TRANSFER,
+    get_global_user_sim_guidelines,
+)
+from tau2.utils.utils import get_now
+
+from ._message_utils import flip_messages, log_messages
+from ._sliding_window import SlidingWindowHistoryProvider
+from ._tau2_utils import convert_agent_framework_messages_to_tau2_messages, convert_tau2_tool_to_function_tool
+
+__all__ = ["ASSISTANT_AGENT_ID", "ORCHESTRATOR_ID", "USER_SIMULATOR_ID", "TaskRunner"]
+
+
+def _get_openai_schema(tool: Any) -> dict[str, Any]:
+    schema = getattr(tool, "openai_schema", None)
+    if isinstance(schema, dict):
+        schema_dict = cast(dict[object, Any], schema)
+        if all(isinstance(key, str) for key in schema_dict):
+            return cast(dict[str, Any], schema_dict)
+    raise TypeError(f"Tool {tool} does not expose a dict openai_schema")
+
+
+# Agent instructions matching tau2's LLMAgent
+ASSISTANT_AGENT_INSTRUCTION = """
+You are a customer service agent that helps the user according to the <policy> provided below.
+In each turn you can either:
+- Send a message to the user.
+- Make a tool call.
+You cannot do both at the same time.
+Try to be helpful and always follow the policy. Always make sure you generate valid JSON only.
+""".strip()
+
+# Default first message from agent (matching tau2)
+DEFAULT_FIRST_AGENT_MESSAGE = "Hi! How can I help you today?"
+
+# Constants of Agent executor IDs
+ASSISTANT_AGENT_ID = "assistant_agent"
+USER_SIMULATOR_ID = "user_simulator"
+ORCHESTRATOR_ID = "orchestrator"
+
+
+class TaskRunner:
+    """Orchestrates task execution using agent framework workflows for tau2 benchmarks.
+
+    Manages conversation flow between assistant agents and user simulators,
+    handles termination conditions, and evaluates performance using tau2 metrics.
+
+    Only "airline" domain is supported for now.
+    """
+
+    # State tracking
+    step_count: int
+    full_conversation: list[Message]
+    termination_reason: TerminationReason | None
+    full_reward_info: RewardInfo | None
+    _final_user_message: list[Message] | None
+    _assistant_executor: AgentExecutor | None
+    _user_executor: AgentExecutor | None
+
+    # Configuration
+    max_steps: int
+    assistant_sampling_temperature: float
+    assistant_window_size: int
+
+    def __init__(self, max_steps: int, assistant_sampling_temperature: float = 0.0, assistant_window_size: int = 32768):
+        """Initialize the TaskRunner.
+
+        Args:
+            max_steps: The maximum number of steps to run.
+            assistant_sampling_temperature: The sampling temperature for the assistant agent.
+            assistant_window_size: The window size for the assistant agent.
+        """
+        self.assistant_sampling_temperature = assistant_sampling_temperature
+        self.assistant_window_size = assistant_window_size
+        self.max_steps = max_steps
+        self.reinit()
+
+    def reinit(self) -> TaskRunner:
+        """Reset all state for a new task run."""
+        self.step_count = 0
+        self.full_conversation = []
+        self.termination_reason = None
+        self.full_reward_info = None
+        self._final_user_message = None
+        self._assistant_executor = None
+        self._user_executor = None
+        logger.info("TaskRunner has been re-initialized.")
+        return self
+
+    def __repr__(self) -> str:
+        """Return string representation of TaskRunner."""
+        return (
+            f"TaskRunner(max_steps={self.max_steps}, step_count={self.step_count}, "
+            f"full_conversation_length={len(self.full_conversation)}, "
+            f"termination_reason={self.termination_reason}, full_reward_info={self.full_reward_info})"
+        )
+
+    def should_not_stop(self, response: AgentExecutorResponse) -> bool:
+        """Based on the response, check whether we should or not stop the conversation."""
+        # Determine who sent this based on executor_id
+        is_from_agent = response.executor_id == ASSISTANT_AGENT_ID
+        is_from_user = response.executor_id == USER_SIMULATOR_ID
+
+        self.step_count += 1
+
+        logger.opt(colors=True).info(
+            f"<bold>[Step {self.step_count}] Received the following response from "
+            f"{'<blue>assistant</blue>' if is_from_agent else '<green>user</green>'}</bold>, "
+            f"routing to {'<green>user</green>' if is_from_agent else '<blue>assistant</blue>'}:"
+        )
+        log_messages(response.agent_response.messages)
+
+        if self.step_count >= self.max_steps:
+            logger.info(f"Max steps ({self.max_steps}) reached - terminating conversation")
+            self.termination_reason = TerminationReason.MAX_STEPS
+            # Terminate the workflow
+            return False
+
+        response_text = response.agent_response.text
+        if is_from_agent and self._is_agent_stop(response_text):
+            logger.info("Agent requested stop - terminating conversation")
+            self.termination_reason = TerminationReason.AGENT_STOP
+            return False
+
+        if is_from_user and self._is_user_stop(response_text):
+            logger.info(f"User requested stop with message: '{response_text}' - terminating conversation")
+            self.termination_reason = TerminationReason.USER_STOP
+            # The final user message won't appear in the assistant's message store,
+            # because it will never arrive there.
+            # We need to store it because it's needed for evaluation.
+            self._final_user_message = flip_messages(response.agent_response.messages)
+            return False
+
+        return True
+
+    def _is_agent_stop(self, _: str) -> bool:
+        """Check if agent wants to stop the conversation."""
+        # Could check for specific stop tokens if agent uses them
+        return False  # Agent doesn't have explicit stop in this setup
+
+    def _is_user_stop(self, text: str) -> bool:
+        """Check if user wants to stop the conversation."""
+        return STOP in text or TRANSFER in text or OUT_OF_SCOPE in text
+
+    def assistant_agent(self, assistant_chat_client: SupportsChatGetResponse) -> Agent:
+        """Create an assistant agent.
+
+        Users can override this method to provide a custom assistant agent.
+
+        Args:
+            assistant_chat_client: The chat client for the assistant agent.
+
+        Returns:
+            The assistant agent.
+        """
+        # Initialize tau2 environment and extract tools/policy
+        # This provides the domain-specific context (airline customer service in this case)
+        env = get_environment()
+        tools = env.get_tools()  # Available actions the assistant can take
+        policy = env.get_policy()  # Guidelines the assistant must follow
+
+        logger.info(
+            f"Environment has {len(env.get_tools())} tools: {', '.join([tool.name for tool in env.get_tools()])}"
+        )
+
+        # Convert tau2 tools to agent framework FunctionTool format
+        # This bridges the gap between tau2's tool system and agent framework's expectations
+        tools = [convert_tau2_tool_to_function_tool(tool) for tool in tools]
+
+        # Combines general customer service behavior with specific policy guidelines
+        assistant_system_prompt = f"""<instructions>
+{ASSISTANT_AGENT_INSTRUCTION}
+</instructions>
+<policy>
+{policy}
+</policy>"""
+
+        # Assistant agent has:
+        # - Access to all domain tools (booking, cancellation, etc.)
+        # - Sliding window memory to handle long conversations within token limits
+        # - Temperature-controlled response generation
+        return Agent(
+            client=assistant_chat_client,
+            instructions=assistant_system_prompt,
+            tools=tools,
+            default_options={"temperature": self.assistant_sampling_temperature},
+            context_providers=[
+                SlidingWindowHistoryProvider(
+                    system_message=assistant_system_prompt,
+                    tool_definitions=[_get_openai_schema(tool) for tool in tools],
+                    max_tokens=self.assistant_window_size,
+                )
+            ],
+        )
+
+    def user_simulator(self, user_simuator_chat_client: SupportsChatGetResponse, task: Task) -> Agent:
+        """Create a user simulator agent.
+
+        Users can override this method to provide a custom user simulator agent.
+
+        Args:
+            user_simuator_chat_client: The chat client for the user simulator agent.
+            task: The task to be executed.
+
+        Returns:
+            The user simulator agent.
+        """
+        # User simulator follows tau2's guidelines for realistic customer behavior
+        # No tools available - users typically don't have direct system access
+        user_sim_guidelines = get_global_user_sim_guidelines(use_tools=False)
+
+        # User simulator prompt combines general guidelines with task-specific scenario
+        user_sim_system_prompt = f"""{user_sim_guidelines}
+<scenario>
+{task.user_scenario.instructions}
+</scenario>"""
+
+        return Agent(
+            client=user_simuator_chat_client,
+            instructions=user_sim_system_prompt,
+            default_options={"temperature": 0.0},
+            # No sliding window for user simulator to maintain full conversation context
+            # TODO(yuge): Consider adding user tools in future for more realistic scenarios
+        )
+
+    async def conversation_orchestrator(
+        self, response: AgentExecutorResponse, ctx: WorkflowContext[AgentExecutorRequest]
+    ) -> None:
+        """Orchestrate conversation flow between assistant and user simulator.
+
+        This is the central routing hub that:
+
+        1. Receives responses from either the assistant agent or user simulator
+        2. Flips message roles to create proper conversation flow (assistant->user, user->assistant)
+        3. Routes the flipped messages to the appropriate target agent
+        4. Maintains the conversation loop until termination conditions are met
+
+        Args:
+            response: The response from either assistant or user simulator agent
+            ctx: Workflow context for sending messages to other executors
+        """
+        # Flip message roles for proper conversation flow
+        # Assistant messages become user messages and vice versa
+        flipped = flip_messages(response.agent_response.messages)
+
+        # Determine source to route to correct target
+        is_from_agent = response.executor_id == ASSISTANT_AGENT_ID
+
+        # Send flipped messages to the opposite agent
+        # Critical: Target ID must be specified to prevent broadcasting to both agents
+        await ctx.send_message(
+            AgentExecutorRequest(messages=flipped, should_respond=True),
+            target_id=USER_SIMULATOR_ID if is_from_agent else ASSISTANT_AGENT_ID,
+        )
+
+    def build_conversation_workflow(self, assistant_agent: Agent, user_simulator_agent: Agent) -> Workflow:
+        """Build the conversation workflow.
+
+        Users can override this method to provide a custom conversation workflow.
+
+        Args:
+            assistant_agent: The assistant agent.
+            user_simulator_agent: The user simulator agent.
+
+        Returns:
+            The conversation workflow.
+        """
+        # STEP 1: Create workflow executors
+        # Each executor wraps an agent or function for workflow orchestration
+        self._assistant_executor = AgentExecutor(assistant_agent, id=ASSISTANT_AGENT_ID)
+        self._user_executor = AgentExecutor(user_simulator_agent, id=USER_SIMULATOR_ID)
+        orchestrator = FunctionExecutor(func=self.conversation_orchestrator, id=ORCHESTRATOR_ID)
+
+        # STEP 2: Build the conversation workflow
+        # Creates a cyclic workflow: Orchestrator -> Assistant -> Orchestrator -> User -> Orchestrator...
+        # The orchestrator acts as a message router that flips roles and routes to appropriate agent
+        return (
+            # Orchestrator manages the conversation flow
+            WorkflowBuilder(max_iterations=10000, start_executor=orchestrator)
+            .add_edge(orchestrator, self._assistant_executor)  # Route messages to assistant
+            .add_edge(
+                self._assistant_executor, orchestrator, condition=self.should_not_stop
+            )  # Check termination after assistant
+            .add_edge(orchestrator, self._user_executor)  # Route messages to user simulator
+            .add_edge(self._user_executor, orchestrator, condition=self.should_not_stop)  # Check termination after user
+            .build()
+        )
+
+    async def run(
+        self,
+        task: Task,
+        assistant_chat_client: SupportsChatGetResponse,
+        user_simulator_chat_client: SupportsChatGetResponse,
+    ) -> list[Message]:
+        """Run a tau2 task using workflow-based agent orchestration.
+
+        This method orchestrates a complex multi-agent simulation:
+
+        1. Sets up tau2 environment and converts tools for agent framework compatibility
+        2. Creates two agents: assistant (with tools) and user simulator (without tools)
+        3. Builds a workflow with orchestrated message routing between agents
+        4. Manages conversation flow until termination conditions are met
+        5. Returns complete conversation history for evaluation
+
+        Args:
+            task: Tau2 task containing scenario, policy, and evaluation criteria
+            assistant_chat_client: LLM client for the assistant agent
+            user_simulator_chat_client: LLM client for the user simulator
+
+        Returns:
+            Complete conversation history as Message list for evaluation
+        """
+        logger.info(f"Starting workflow agent for task {task.id}: {task.description.purpose}")  # type: ignore[unused-ignore]
+        logger.info(f"Assistant chat client: {assistant_chat_client}")
+        logger.info(f"User simulator chat client: {user_simulator_chat_client}")
+
+        # STEP 1: Create agents
+        assistant_agent = self.assistant_agent(assistant_chat_client)
+        user_simulator_agent = self.user_simulator(user_simulator_chat_client, task)
+
+        # STEP 2: Create the conversation workflow
+        workflow = self.build_conversation_workflow(assistant_agent, user_simulator_agent)
+
+        # STEP 3: Initialize conversation with standard greeting
+        # Matches tau2's expected conversation start pattern
+        logger.info(f"Starting workflow with hardcoded greeting: '{DEFAULT_FIRST_AGENT_MESSAGE}'")
+
+        first_message = Message(role="assistant", contents=[DEFAULT_FIRST_AGENT_MESSAGE])
+        initial_greeting = AgentExecutorResponse(
+            executor_id=ASSISTANT_AGENT_ID,
+            agent_response=AgentResponse(messages=[first_message]),
+            full_conversation=[Message(role="assistant", contents=[DEFAULT_FIRST_AGENT_MESSAGE])],
+        )
+
+        # STEP 4: Execute the workflow and collect results
+        # The workflow runs until termination conditions are met (max steps, stop signals, etc.)
+        await workflow.run(initial_greeting)
+
+        # STEP 5: Ensemble the conversation history needed for evaluation.
+        # It's coming from three parts:
+        # 1. The initial greeting
+        # 2. The assistant's session state (full history, not just the truncated window)
+        # 3. The final user message (if any)
+        session_state: dict[str, Any] = self._assistant_executor._session.state  # type: ignore
+        all_messages: list[Message] = list(
+            session_state.get(InMemoryHistoryProvider.DEFAULT_SOURCE_ID, {}).get("messages", [])
+        )
+        full_conversation = [first_message, *all_messages]
+        if self._final_user_message is not None:
+            full_conversation.extend(self._final_user_message)
+
+        logger.opt(colors=True).info(
+            f"<green>WORKFLOW COMPLETED WITH {len(full_conversation)} MESSAGES. "
+            f"Termination reason: {self.termination_reason}.</green>"
+        )
+        log_messages(full_conversation)
+
+        return full_conversation
+
+    def evaluate(
+        self, task_input: Task, conversation: list[Message], termination_reason: TerminationReason | None
+    ) -> float:
+        """Evaluate agent performance using tau2's comprehensive evaluation system.
+
+        Bridges agent framework conversation results with tau2's evaluation pipeline.
+        Considers task completion, policy adherence, conversation quality, and tool usage.
+
+        Args:
+            task_input: Original tau2 task containing evaluation criteria
+            conversation: Complete conversation history from workflow execution
+            termination_reason: How/why the conversation ended (affects scoring)
+
+        Returns:
+            Numeric reward score (0.0-1.0) representing overall performance
+
+        Side Effects:
+            Stores detailed evaluation results in self.full_reward_info
+        """
+        # Handle missing termination reason (can happen with unexpected workflow endings)
+        if termination_reason is None:
+            termination_reason = TerminationReason.TOO_MANY_ERRORS
+
+        # Convert agent framework ChatMessages to tau2 Message format for evaluation
+        tau2_messages = convert_agent_framework_messages_to_tau2_messages(conversation)
+
+        # Package conversation and metadata for tau2's evaluation system
+        simulation = SimulationRun(
+            id=str(uuid.uuid4()),  # Unique identifier for this evaluation run
+            task_id=task_input.id,  # Links evaluation back to original task
+            start_time=get_now(),  # Timestamp for evaluation records
+            end_time=get_now(),  # Duration is 0 since this is post-hoc evaluation
+            duration=0.0,
+            termination_reason=termination_reason,  # Context for how conversation ended
+            messages=tau2_messages,  # The actual conversation to evaluate
+        )
+
+        # Run comprehensive multi-dimensional evaluation
+        # EvaluationType.ALL: evaluates task completion, policy adherence, conversation quality, ...
+        # solo_mode=False: indicates multi-agent conversation (assistant + user simulator)
+        self.full_reward_info = evaluate_simulation(
+            simulation=simulation,
+            task=task_input,
+            evaluation_type=EvaluationType.ALL,
+            solo_mode=False,
+            domain="airline",
+        )
+
+        logger.info(
+            f"Evaluation completed - Reward: {self.full_reward_info.reward if self.full_reward_info else None}, "
+            f"Info: {self.full_reward_info}"
+        )
+        return self.full_reward_info.reward if self.full_reward_info else 0.0

python/packages/lab/tau2/samples/.env.example

@@ -0,0 +1,4 @@
+TAU2_DATA_DIR=/path/to/your/data
+
+OPENAI_API_KEY=dummy
+OPENAI_BASE_URL=http://127.0.0.1:12345/

python/packages/lab/tau2/samples/run_benchmark.py

@@ -0,0 +1,279 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import argparse
+import asyncio
+import json
+import os
+import traceback
+from datetime import datetime
+from typing import Any
+
+from agent_framework.lab.tau2 import TaskRunner, patch_env_set_state
+from agent_framework.openai import OpenAIChatClient
+from loguru import logger
+from tau2.domains.airline.environment import get_tasks
+
+
+def to_dumpable(result: dict[str, Any]) -> dict[str, Any]:
+    """Convert benchmark result to JSONL-serializable format.
+
+    Handles both successful runs and error cases, ensuring consistent output
+    format for downstream analysis. Converts Pydantic models to dictionaries
+    and extracts enum values for JSON compatibility.
+    """
+    if "error" in result:
+        # Error case: minimal structure with zero reward
+        return {
+            "id": result["task"].id,
+            "error": result["error"],
+            "evaluation": {
+                "reward": 0.0,  # Standard zero reward for failed runs
+            },
+            "config": result["config"],
+            "task": result["task"].model_dump(),
+        }
+    # Success case: full result structure
+    return {
+        "id": result["task"].id,
+        "evaluation": result["evaluation"].model_dump(),  # Detailed evaluation metrics
+        "config": result["config"],  # Model configuration used
+        "termination_reason": result["termination_reason"].value,  # Enum to string
+        "messages": [m.model_dump() for m in result["messages"]],  # Full conversation
+        "task": result["task"].model_dump(),  # Task specification
+    }
+
+
+async def run_benchmark(assistant_model: str, user_model: str, debug_task_id: str | None, max_steps: int):
+    """Run comprehensive tau2 benchmark evaluation using agent framework.
+
+    This is the main function that:
+
+    1. Sets up output file handling (full benchmark vs debug mode)
+    2. Loads tau2 task dataset and configures LLM clients
+    3. Runs each task through the agent framework workflow
+    4. Evaluates performance using tau2's multi-dimensional metrics
+    5. Aggregates results and calculates overall benchmark scores
+
+    Args:
+        assistant_model: Model ID for the customer service agent (e.g., "gpt-4o")
+        user_model: Model ID for the user simulator (e.g., "gpt-4o")
+        debug_task_id: Optional specific task ID to run (disables batch processing)
+        max_steps: Maximum conversation steps before forced termination
+
+    Output:
+        Creates timestamped JSONL file with detailed results for analysis
+        Prints summary statistics to console with colored logging
+    """
+    # STEP 1: Configure output handling based on execution mode
+    result_filename = None
+    if debug_task_id is None:
+        # Full benchmark mode: create timestamped results file
+        timestamp = datetime.now().strftime("%m%d%H%M")  # Format: MMDDHHMM
+        result_filename = f"results/{assistant_model}_user-{user_model}_{timestamp}.jsonl"
+        os.makedirs("results", exist_ok=True)
+        logger.info(f"Results will be saved to: {result_filename}")
+    else:
+        # Debug mode: single task, no file output, verbose logging
+        logger.info(f"Debug mode: targeting task ID {debug_task_id}")
+
+    # STEP 2: Load tau2 dataset and validate environment
+    tasks = get_tasks()  # Loads all tau2 airline customer service tasks
+    logger.info(f"Found {len(tasks)} tasks in the dataset")
+
+    logger_ = logger.opt(colors=True)  # Enable colored console output
+
+    # Validate required OpenAI configuration
+    # Both models use the same endpoint but can be different model types
+    openai_base_url = os.getenv("OPENAI_BASE_URL")
+    if openai_base_url is None:
+        raise ValueError("OPENAI_BASE_URL must be set")
+    openai_api_key = os.getenv("OPENAI_API_KEY")
+    if openai_api_key is None:
+        raise ValueError("OPENAI_API_KEY must be set")
+
+    # STEP 3: Initialize LLM clients for both agent roles
+    # Assistant: handles customer service with access to tools and policies
+    assistant_chat_client = OpenAIChatClient(
+        base_url=openai_base_url,
+        api_key=openai_api_key,
+        model=assistant_model,
+    )
+
+    # User simulator: simulates realistic customer behavior and requests
+    user_simulator_chat_client = OpenAIChatClient(
+        base_url=openai_base_url,
+        api_key=openai_api_key,
+        model=user_model,
+    )
+
+    # STEP 4: Filter task set for debug mode
+    if debug_task_id is not None:
+        tasks = [task for task in tasks if task.id == debug_task_id]
+        if not tasks:
+            logger.error(f"Task ID {debug_task_id} not found in dataset")
+            return
+
+    # STEP 5: Initialize evaluation tracking
+    all_rewards: list[float] = []  # Stores reward scores for final statistics
+    task_runner = TaskRunner(max_steps=max_steps)  # Reusable workflow orchestrator
+
+    # STEP 6: Execute benchmark across all tasks with proper file handling
+    def write_result(result_fp, result):
+        """Write result to file if file pointer is provided."""
+        if result_fp is not None:
+            result_fp.write(json.dumps(to_dumpable(result), default=str) + "\n")
+
+    # Use context manager for file handling
+    if result_filename:
+        with open(result_filename, "a") as result_fp:
+            for task in tasks:
+                logger_.info(f"<red>Testing task #{task.id}</red>")
+                logger_.info(f"<cyan>Purpose:</cyan> {task.description.purpose}")  # type: ignore
+
+                # Initialize result structure for this task
+                result: dict[str, Any] = {
+                    "config": {
+                        "assistant": assistant_chat_client.model,
+                        "user": user_simulator_chat_client.model,
+                    },
+                    "task": task,
+                }
+
+                # Log user scenario context for transparency
+                if task.user_scenario and task.user_scenario.instructions:
+                    logger_.info(f"<cyan>User scenario:</cyan> {task.user_scenario.instructions.reason_for_call}")  # type: ignore
+
+                try:
+                    # Execute the workflow: agent + user simulator conversation
+                    conversation = await task_runner.run(task, assistant_chat_client, user_simulator_chat_client)
+
+                    # Evaluate performance using tau2's comprehensive metrics
+                    reward_value = task_runner.evaluate(task, conversation, task_runner.termination_reason)
+
+                    # Store detailed results for analysis
+                    result["evaluation"] = task_runner.full_reward_info  # Full evaluation breakdown
+                    result["messages"] = conversation  # Complete conversation history
+                    result["termination_reason"] = task_runner.termination_reason  # How conversation ended
+
+                    # Log evaluation results (escape HTML for colored output)
+                    reward_str = str(task_runner.full_reward_info).replace("<", r"\<")
+                    logger_.info(f"<cyan>Final evaluation:</cyan> {reward_str}")
+
+                except Exception as e:
+                    # Robust error handling: capture all failures for analysis
+                    logger_.error(f"<red>Error testing task #{task.id}:</red> {e}")
+                    result["error"] = traceback.format_exc()  # Full stack trace for debugging
+
+                    traceback.print_exc()  # Console output for immediate debugging
+                    reward_value = 0.0  # Zero score for failed runs
+
+                # STEP 7: Persist results incrementally (enables partial analysis)
+                write_result(result_fp, result)
+
+                all_rewards.append(reward_value)  # Track for final statistics
+
+                # Reset runner state for next task
+                task_runner.reinit()
+    else:
+        # Debug mode without file output
+        for task in tasks:
+            logger_.info(f"<red>Testing task #{task.id}</red>")
+            logger_.info(f"<cyan>Purpose:</cyan> {task.description.purpose}")  # type: ignore
+
+            # Initialize result structure for this task
+            result: dict[str, Any] = {
+                "config": {
+                    "assistant": assistant_chat_client.model,
+                    "user": user_simulator_chat_client.model,
+                },
+                "task": task,
+            }
+
+            # Log user scenario context for transparency
+            if task.user_scenario and task.user_scenario.instructions:
+                logger_.info(f"<cyan>User scenario:</cyan> {task.user_scenario.instructions.reason_for_call}")  # type: ignore
+
+            try:
+                # Execute the workflow: agent + user simulator conversation
+                conversation = await task_runner.run(task, assistant_chat_client, user_simulator_chat_client)
+
+                # Evaluate performance using tau2's comprehensive metrics
+                reward_value = task_runner.evaluate(task, conversation, task_runner.termination_reason)
+
+                # Log evaluation results (escape HTML for colored output)
+                reward_str = str(task_runner.full_reward_info).replace("<", r"\<")
+                logger_.info(f"<cyan>Final evaluation:</cyan> {reward_str}")
+
+            except Exception as e:
+                # Robust error handling: capture all failures for analysis
+                logger_.error(f"<red>Error testing task #{task.id}:</red> {e}")
+                traceback.print_exc()  # Console output for immediate debugging
+                reward_value = 0.0  # Zero score for failed runs
+
+            all_rewards.append(reward_value)  # Track for final statistics
+
+            # Reset runner state for next task
+            task_runner.reinit()
+
+    # STEP 8: Calculate overall benchmark performance and report final statistics
+    all_accuracy = sum(all_rewards) / len(all_rewards) if all_rewards else 0.0
+
+    # Report final statistics with colored formatting
+    logger_.info("<green>Final Results:</green>")
+    logger_.info(f"<cyan>All tasks accuracy:</cyan> {all_accuracy:.2f} ({int(sum(all_rewards))}/{len(tasks)})")
+
+
+if __name__ == "__main__":
+    """Command-line interface for tau2 benchmark execution.
+
+    Provides flexible execution modes:
+
+    - Full benchmark: Runs all tasks and generates timestamped results file
+    - Debug mode: Single task execution with verbose logging for development
+    - Environment patching: Optional compatibility layer for tau2-bench integration
+
+    Usage Examples:
+        # Full benchmark with default models
+        python run_benchmark.py
+
+        # Custom models
+        python run_benchmark.py --assistant gpt-4o --user gpt-4o-mini
+
+        # Debug specific task
+        python run_benchmark.py --debug-task-id task_123
+
+        # Disable environment patching for testing
+        python run_benchmark.py --disable-env-patch
+    """
+
+    parser = argparse.ArgumentParser(description="Run tau2-agent-framework model test")
+
+    # Model configuration arguments
+    parser.add_argument("--assistant", type=str, default="gpt-4.1", help="Assistant model id, e.g., gpt-4.1-mini")
+    parser.add_argument("--user", type=str, default="gpt-4.1", help="User model id")
+
+    # Execution mode arguments
+    parser.add_argument(
+        "--debug-task-id", type=str, default=None, help="Debug a specific task ID (disables result file creation)"
+    )
+    parser.add_argument("--max-steps", type=int, default=100, help="Maximum number of steps to run")
+
+    # Environment configuration arguments
+    parser.add_argument("--disable-env-patch", action="store_true", help="Disable patching tau2-bench environment")
+
+    args = parser.parse_args()
+
+    # Apply environment patch for tau2-bench compatibility
+    # This modifies tau2's environment to be more flexible with tool call validation
+    if not args.disable_env_patch:
+        patch_env_set_state()
+
+    # Execute benchmark with configured parameters
+    asyncio.run(
+        run_benchmark(
+            assistant_model=args.assistant,
+            user_model=args.user,
+            debug_task_id=args.debug_task_id,
+            max_steps=args.max_steps,
+        )
+    )

python/packages/lab/tau2/tests/test_message_utils.py

@@ -0,0 +1,283 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+from unittest.mock import patch
+
+import pytest
+
+try:
+    from litellm import completion as _litellm_completion  # noqa: F401
+except Exception:
+    pytest.skip("LiteLLM import surface required by tau2 is unavailable.", allow_module_level=True)
+
+from agent_framework._types import Content, Message
+from agent_framework_lab_tau2._message_utils import flip_messages, log_messages
+
+
+def test_flip_messages_user_to_assistant():
+    """Test flipping user message to assistant."""
+    messages = [
+        Message(
+            role="user",
+            contents=[Content.from_text(text="Hello assistant")],
+            author_name="User1",
+            message_id="msg_001",
+        )
+    ]
+
+    flipped = flip_messages(messages)
+
+    assert len(flipped) == 1
+    assert flipped[0].role == "assistant"
+    assert flipped[0].text == "Hello assistant"
+    assert flipped[0].author_name == "User1"
+    assert flipped[0].message_id == "msg_001"
+
+
+def test_flip_messages_assistant_to_user():
+    """Test flipping assistant message to user."""
+    messages = [
+        Message(
+            role="assistant",
+            contents=[Content.from_text(text="Hello user")],
+            author_name="Assistant1",
+            message_id="msg_002",
+        )
+    ]
+
+    flipped = flip_messages(messages)
+
+    assert len(flipped) == 1
+    assert flipped[0].role == "user"
+    assert flipped[0].text == "Hello user"
+    assert flipped[0].author_name == "Assistant1"
+    assert flipped[0].message_id == "msg_002"
+
+
+def test_flip_messages_assistant_with_function_calls_filtered():
+    """Test that function calls are filtered out when flipping assistant to user."""
+    function_call = Content.from_function_call(call_id="call_123", name="test_function", arguments={"param": "value"})
+
+    messages = [
+        Message(
+            role="assistant",
+            contents=[
+                Content.from_text(text="I'll call a function"),
+                function_call,
+                Content.from_text(text="After the call"),
+            ],
+            message_id="msg_003",
+        )
+    ]
+
+    flipped = flip_messages(messages)
+
+    assert len(flipped) == 1
+    assert flipped[0].role == "user"
+    # Function call should be filtered out
+    assert len(flipped[0].contents) == 2
+    assert all(content.type == "text" for content in flipped[0].contents)
+    assert "I'll call a function" in flipped[0].text
+    assert "After the call" in flipped[0].text
+
+
+def test_flip_messages_assistant_with_only_function_calls_skipped():
+    """Test that assistant messages with only function calls are skipped."""
+    function_call = Content.from_function_call(call_id="call_456", name="another_function", arguments={"key": "value"})
+
+    messages = [
+        Message(role="assistant", contents=[function_call], message_id="msg_004")  # Only function call, no text
+    ]
+
+    flipped = flip_messages(messages)
+
+    # Should be empty since the message had no text content after filtering
+    assert len(flipped) == 0
+
+
+def test_flip_messages_tool_messages_skipped():
+    """Test that tool messages are skipped."""
+    function_result = Content.from_function_result(call_id="call_789", result={"success": True})
+
+    messages = [Message(role="tool", contents=[function_result])]
+
+    flipped = flip_messages(messages)
+
+    # Tool messages should be skipped
+    assert len(flipped) == 0
+
+
+def test_flip_messages_system_messages_preserved():
+    """Test that system messages are preserved as-is."""
+    messages = [Message(role="system", contents=[Content.from_text(text="System instruction")], message_id="sys_001")]
+
+    flipped = flip_messages(messages)
+
+    assert len(flipped) == 1
+    assert flipped[0].role == "system"
+    assert flipped[0].text == "System instruction"
+    assert flipped[0].message_id == "sys_001"
+
+
+def test_flip_messages_mixed_conversation():
+    """Test flipping a mixed conversation."""
+    function_call = Content.from_function_call(call_id="call_mixed", name="mixed_function", arguments={})
+
+    function_result = Content.from_function_result(call_id="call_mixed", result="function result")
+
+    messages = [
+        Message(role="system", contents=[Content.from_text(text="System prompt")]),
+        Message(role="user", contents=[Content.from_text(text="User question")]),
+        Message(role="assistant", contents=[Content.from_text(text="Assistant response"), function_call]),
+        Message(role="tool", contents=[function_result]),
+        Message(role="assistant", contents=[Content.from_text(text="Final response")]),
+    ]
+
+    flipped = flip_messages(messages)
+
+    # Should have: system (unchanged), assistant (from user), user (from assistant, filtered),
+    # assistant (from final assistant)
+    assert len(flipped) == 4
+
+    # Check each flipped message
+    assert flipped[0].role == "system"
+    assert flipped[0].text == "System prompt"
+
+    assert flipped[1].role == "assistant"
+    assert flipped[1].text == "User question"
+
+    assert flipped[2].role == "user"
+    assert flipped[2].text == "Assistant response"  # Function call filtered out
+
+    # Tool message skipped
+
+    assert flipped[3].role == "user"
+    assert flipped[3].text == "Final response"
+
+
+def test_flip_messages_empty_list():
+    """Test flipping empty message list."""
+    messages: list[Message] = []
+    flipped = flip_messages(messages)
+    assert len(flipped) == 0
+
+
+def test_flip_messages_preserves_metadata():
+    """Test that message metadata is preserved during flipping."""
+    messages = [
+        Message(
+            role="user",
+            contents=[Content.from_text(text="Test message")],
+            author_name="TestUser",
+            message_id="test_123",
+        )
+    ]
+
+    flipped = flip_messages(messages)
+
+    assert len(flipped) == 1
+    assert flipped[0].author_name == "TestUser"
+    assert flipped[0].message_id == "test_123"
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_text_content(mock_logger):
+    """Test logging messages with text content."""
+    messages = [
+        Message(role="user", contents=[Content.from_text(text="Hello")]),
+        Message(role="assistant", contents=[Content.from_text(text="Hi there!")]),
+    ]
+
+    log_messages(messages)
+
+    # Should have called logger.info for each message
+    assert mock_logger.opt.return_value.info.call_count == 2
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_function_call(mock_logger):
+    """Test logging messages with function calls."""
+    function_call = Content.from_function_call(call_id="call_log", name="log_function", arguments={"param": "value"})
+
+    messages = [Message(role="assistant", contents=[function_call])]
+
+    log_messages(messages)
+
+    # Should log the function call
+    mock_logger.opt.return_value.info.assert_called()
+    call_args = mock_logger.opt.return_value.info.call_args[0][0]
+    assert "TOOL_CALL" in call_args
+    assert "log_function" in call_args
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_function_result(mock_logger):
+    """Test logging messages with function results."""
+    function_result = Content.from_function_result(call_id="call_result", result="success")
+
+    messages = [Message(role="tool", contents=[function_result])]
+
+    log_messages(messages)
+
+    # Should log the function result
+    mock_logger.opt.return_value.info.assert_called()
+    call_args = mock_logger.opt.return_value.info.call_args[0][0]
+    assert "TOOL_RESULT" in call_args
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_different_roles(mock_logger):
+    """Test logging messages with different roles get different colors."""
+    messages = [
+        Message(role="system", contents=[Content.from_text(text="System")]),
+        Message(role="user", contents=[Content.from_text(text="User")]),
+        Message(role="assistant", contents=[Content.from_text(text="Assistant")]),
+        Message(role="tool", contents=[Content.from_text(text="Tool")]),
+    ]
+
+    log_messages(messages)
+
+    # Should have called logger for each message
+    assert mock_logger.opt.return_value.info.call_count == 4
+
+    # Check that different color tags are used
+    calls = mock_logger.opt.return_value.info.call_args_list
+    system_call = calls[0][0][0]
+    user_call = calls[1][0][0]
+    assistant_call = calls[2][0][0]
+    tool_call = calls[3][0][0]
+
+    assert "cyan" in system_call or "SYSTEM" in system_call
+    assert "green" in user_call or "USER" in user_call
+    assert "blue" in assistant_call or "ASSISTANT" in assistant_call
+    assert "yellow" in tool_call or "TOOL" in tool_call
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_escapes_html(mock_logger):
+    """Test that HTML-like characters are properly escaped in log output."""
+    messages = [Message(role="user", contents=[Content.from_text(text="Message with <tag> content")])]
+
+    log_messages(messages)
+
+    mock_logger.opt.return_value.info.assert_called()
+    call_args = mock_logger.opt.return_value.info.call_args[0][0]
+    # Should escape < characters
+    assert "\\<tag>" in call_args or "&lt;tag&gt;" in call_args
+
+
+@patch("agent_framework_lab_tau2._message_utils.logger")
+def test_log_messages_mixed_content_types(mock_logger):
+    """Test logging messages with mixed content types."""
+    function_call = Content.from_function_call(call_id="mixed_call", name="mixed_function", arguments={"key": "value"})
+
+    messages = [
+        Message(
+            role="assistant",
+            contents=[Content.from_text(text="I'll call a function"), function_call, Content.from_text(text="Done!")],
+        )
+    ]
+
+    log_messages(messages)
+
+    # Should log multiple times for different content types
+    assert mock_logger.opt.return_value.info.call_count == 3

python/packages/lab/tau2/tests/test_sliding_window.py

@@ -0,0 +1,213 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for sliding window history provider."""
+
+from unittest.mock import patch
+
+import pytest
+
+try:
+    from litellm import completion as _litellm_completion  # noqa: F401
+except Exception:
+    pytest.skip("LiteLLM import surface required by tau2 is unavailable.", allow_module_level=True)
+
+from agent_framework import InMemoryHistoryProvider
+from agent_framework._types import Content, Message
+from agent_framework_lab_tau2._sliding_window import SlidingWindowHistoryProvider
+
+
+def _make_state(provider: SlidingWindowHistoryProvider, messages: list[Message] | None = None) -> dict:
+    """Helper to create a session state dict with messages pre-loaded."""
+    state: dict = {}
+    if messages:
+        state["messages"] = list(messages)
+    return state
+
+
+def test_initialization():
+    """Test initializing with parameters."""
+    provider = SlidingWindowHistoryProvider(
+        max_tokens=2000,
+        system_message="You are a helpful assistant",
+        tool_definitions=[{"name": "test_tool"}],
+    )
+
+    assert provider.max_tokens == 2000
+    assert provider.system_message == "You are a helpful assistant"
+    assert provider.tool_definitions == [{"name": "test_tool"}]
+    assert provider.source_id == InMemoryHistoryProvider.DEFAULT_SOURCE_ID
+
+
+async def test_get_messages_empty():
+    """Test getting messages from empty state."""
+    provider = SlidingWindowHistoryProvider(max_tokens=1000)
+    messages = await provider.get_messages(None, state={})
+    assert messages == []
+
+
+async def test_get_messages_simple():
+    """Test getting messages without truncation."""
+    provider = SlidingWindowHistoryProvider(max_tokens=10000)
+    msgs = [
+        Message(role="user", contents=[Content.from_text(text="What's the weather?")]),
+        Message(role="assistant", contents=[Content.from_text(text="I can help with that.")]),
+    ]
+    state = _make_state(provider, msgs)
+
+    result = await provider.get_messages(None, state=state)
+    assert len(result) == 2
+    assert result[0].text == "What's the weather?"
+    assert result[1].text == "I can help with that."
+
+
+async def test_save_and_get_messages():
+    """Test saving then getting messages with truncation."""
+    provider = SlidingWindowHistoryProvider(max_tokens=50)
+    state: dict = {}
+
+    # Save many messages
+    msgs = [
+        Message(role="user", contents=[Content.from_text(text=f"Message {i} with some content")]) for i in range(10)
+    ]
+    await provider.save_messages(None, msgs, state=state)
+
+    # get_messages returns truncated
+    truncated = await provider.get_messages(None, state=state)
+    # Full history is in session state
+    all_msgs = state["messages"]
+
+    assert len(all_msgs) == 10
+    assert len(truncated) < len(all_msgs)
+
+
+def test_get_token_count_basic():
+    """Test basic token counting."""
+    provider = SlidingWindowHistoryProvider(max_tokens=1000)
+    messages = [Message(role="user", contents=[Content.from_text(text="Hello")])]
+
+    token_count = provider._get_token_count(messages)
+    assert token_count > 0
+
+
+def test_get_token_count_with_system_message():
+    """Test token counting includes system message."""
+    provider = SlidingWindowHistoryProvider(max_tokens=1000, system_message="You are a helpful assistant")
+
+    count_empty = provider._get_token_count([])
+    count_with_msg = provider._get_token_count([Message(role="user", contents=[Content.from_text(text="Hello")])])
+
+    assert count_with_msg > count_empty
+    assert count_empty > 0  # System message contributes tokens
+
+
+def test_get_token_count_function_call():
+    """Test token counting with function calls."""
+    function_call = Content.from_function_call(call_id="call_123", name="test_function", arguments={"param": "value"})
+    provider = SlidingWindowHistoryProvider(max_tokens=1000)
+
+    token_count = provider._get_token_count([Message(role="assistant", contents=[function_call])])
+    assert token_count > 0
+
+
+def test_get_token_count_function_result():
+    """Test token counting with function results."""
+    function_result = Content.from_function_result(call_id="call_123", result={"success": True, "data": "result"})
+    provider = SlidingWindowHistoryProvider(max_tokens=1000)
+
+    token_count = provider._get_token_count([Message(role="tool", contents=[function_result])])
+    assert token_count > 0
+
+
+@patch("agent_framework_lab_tau2._sliding_window.logger")
+def test_truncate_removes_old_messages(mock_logger):
+    """Test that truncation removes old messages when token limit exceeded."""
+    provider = SlidingWindowHistoryProvider(max_tokens=20)
+
+    messages = [
+        Message(
+            role="user",
+            contents=[Content.from_text(text="This is a very long message that should exceed the token limit")],
+        ),
+        Message(
+            role="assistant",
+            contents=[
+                Content.from_text(text="This is another very long message that should also exceed the token limit")
+            ],
+        ),
+        Message(role="user", contents=[Content.from_text(text="Short msg")]),
+    ]
+
+    result = provider._truncate(list(messages))
+    assert len(result) < len(messages)
+    assert mock_logger.warning.called
+
+
+@patch("agent_framework_lab_tau2._sliding_window.logger")
+def test_truncate_removes_leading_tool_messages(mock_logger):
+    """Test that truncation removes leading tool messages."""
+    provider = SlidingWindowHistoryProvider(max_tokens=10000)
+
+    tool_message = Message(role="tool", contents=[Content.from_function_result(call_id="call_123", result="result")])
+    user_message = Message(role="user", contents=[Content.from_text(text="Hello")])
+
+    result = provider._truncate([tool_message, user_message])
+    assert len(result) == 1
+    assert result[0].role == "user"
+    mock_logger.warning.assert_called()
+
+
+def test_estimate_any_object_token_count():
+    """Test token counting for various object types."""
+    provider = SlidingWindowHistoryProvider(max_tokens=1000)
+
+    assert provider._estimate_any_object_token_count({"key": "value"}) > 0
+    assert provider._estimate_any_object_token_count("test string") > 0
+
+    # Non-serializable falls back to str()
+    class Custom:
+        def __str__(self):
+            return "Custom instance"
+
+    assert provider._estimate_any_object_token_count(Custom()) > 0
+
+
+async def test_real_world_scenario():
+    """Test a realistic conversation scenario."""
+    provider = SlidingWindowHistoryProvider(max_tokens=30, system_message="You are a helpful assistant")
+    state: dict = {}
+
+    conversation = [
+        Message(role="user", contents=[Content.from_text(text="Hello, how are you?")]),
+        Message(
+            role="assistant",
+            contents=[Content.from_text(text="I'm doing well, thank you! How can I help you today?")],
+        ),
+        Message(role="user", contents=[Content.from_text(text="Can you tell me about the weather?")]),
+        Message(
+            role="assistant",
+            contents=[
+                Content.from_text(
+                    text="I'd be happy to help with weather information, "
+                    "but I don't have access to current weather data."
+                )
+            ],
+        ),
+        Message(role="user", contents=[Content.from_text(text="What about telling me a joke instead?")]),
+        Message(
+            role="assistant",
+            contents=[
+                Content.from_text(text="Sure! Why don't scientists trust atoms? Because they make up everything!")
+            ],
+        ),
+    ]
+
+    await provider.save_messages(None, conversation, state=state)
+
+    truncated = await provider.get_messages(None, state=state)
+    all_msgs = state["messages"]
+
+    assert len(all_msgs) == 6
+    assert len(truncated) <= 6
+
+    token_count = provider._get_token_count(truncated)
+    assert token_count <= provider.max_tokens * 1.1

python/packages/lab/tau2/tests/test_tau2_utils.py

@@ -0,0 +1,212 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+"""Tests for tau2 utils module."""
+
+import pytest
+
+try:
+    from litellm import completion as _litellm_completion  # noqa: F401
+except Exception:
+    pytest.skip("LiteLLM import surface required by tau2 is unavailable.", allow_module_level=True)
+
+from agent_framework import Content, FunctionTool, Message
+from agent_framework_lab_tau2._tau2_utils import (
+    convert_agent_framework_messages_to_tau2_messages,
+    convert_tau2_tool_to_function_tool,
+)
+from pydantic import BaseModel
+from tau2.data_model.message import AssistantMessage, SystemMessage, ToolCall, ToolMessage, UserMessage
+
+
+class _DummyToolInput(BaseModel):
+    param: str
+
+
+class _DummyToolResult(BaseModel):
+    output: str
+
+
+class _DummyTau2Tool:
+    def __init__(self, name: str, description: str) -> None:
+        self.name = name
+        self._description = description
+        self.params = _DummyToolInput
+
+    def _get_description(self) -> str:
+        return self._description
+
+    def __call__(self, **kwargs: str) -> _DummyToolResult:
+        return _DummyToolResult(output=kwargs["param"])
+
+
+def test_convert_tau2_tool_to_function_tool_basic():
+    """Test basic conversion from tau2 tool to FunctionTool."""
+    tau2_tool = _DummyTau2Tool(name="lookup_booking", description="Lookup booking by id.")
+
+    # Convert the tool
+    tool = convert_tau2_tool_to_function_tool(tau2_tool)  # ty: ignore[invalid-argument-type]  # pyrefly: ignore[bad-argument-type]  # pyright: ignore[reportArgumentType]
+
+    # Verify the conversion
+    assert isinstance(tool, FunctionTool)
+    assert tool.name == tau2_tool.name
+    assert tool.description == tau2_tool._get_description()
+    assert tool.input_model == tau2_tool.params
+
+    assert tool.func is not None
+    result = tool.func(param="ABC123")
+    assert isinstance(result, _DummyToolResult)
+    assert result.output == "ABC123"
+    assert callable(tool.func)
+
+
+def test_convert_tau2_tool_to_function_tool_multiple_tools():
+    """Test conversion with multiple tau2 tools."""
+    tools = [
+        _DummyTau2Tool(name="lookup_booking", description="Lookup booking by id."),
+        _DummyTau2Tool(name="cancel_booking", description="Cancel an existing booking."),
+        _DummyTau2Tool(name="check_policy", description="Get policy details."),
+    ]
+
+    # Convert multiple tools
+    function_tools = [convert_tau2_tool_to_function_tool(tool) for tool in tools]  # ty: ignore[invalid-argument-type]  # pyrefly: ignore[bad-argument-type]  # pyright: ignore[reportArgumentType]
+
+    # Verify all conversions
+    for tool, tau2_tool in zip(function_tools, tools, strict=False):
+        assert isinstance(tool, FunctionTool)
+        assert tool.name == tau2_tool.name
+        assert tool.description == tau2_tool._get_description()
+        assert tool.input_model == tau2_tool.params
+        assert callable(tool.func)
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_system():
+    """Test converting system message."""
+    messages = [Message(role="system", contents=[Content.from_text(text="System instruction")])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], SystemMessage)
+    assert tau2_messages[0].role == "system"
+    assert tau2_messages[0].content == "System instruction"
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_user():
+    """Test converting user message."""
+    messages = [Message(role="user", contents=[Content.from_text(text="Hello assistant")])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], UserMessage)
+    assert tau2_messages[0].role == "user"
+    assert tau2_messages[0].content == "Hello assistant"
+    assert tau2_messages[0].tool_calls is None
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_assistant():
+    """Test converting assistant message."""
+    messages = [Message(role="assistant", contents=[Content.from_text(text="Hello user")])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], AssistantMessage)
+    assert tau2_messages[0].role == "assistant"
+    assert tau2_messages[0].content == "Hello user"
+    assert tau2_messages[0].tool_calls is None
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_with_function_call():
+    """Test converting message with function call."""
+    function_call = Content.from_function_call(call_id="call_123", name="test_function", arguments={"param": "value"})
+
+    messages = [Message(role="assistant", contents=[Content.from_text(text="I'll call a function"), function_call])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], AssistantMessage)
+    assert tau2_messages[0].content == "I'll call a function"
+    assert tau2_messages[0].tool_calls is not None
+    assert len(tau2_messages[0].tool_calls) == 1
+
+    tool_call = tau2_messages[0].tool_calls[0]
+    assert isinstance(tool_call, ToolCall)
+    assert tool_call.id == "call_123"
+    assert tool_call.name == "test_function"
+    assert tool_call.arguments == {"param": "value"}
+    assert tool_call.requestor == "assistant"
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_with_function_result():
+    """Test converting message with function result."""
+    function_result = Content.from_function_result(call_id="call_123", result={"success": True, "data": "result data"})
+
+    messages = [Message(role="tool", contents=[function_result])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], ToolMessage)
+    assert tau2_messages[0].id == "call_123"
+    assert tau2_messages[0].role == "tool"
+    assert tau2_messages[0].content is not None
+    assert '{"success": true, "data": "result data"}' in tau2_messages[0].content
+    assert tau2_messages[0].requestor == "assistant"
+    assert tau2_messages[0].error is False
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_with_error():
+    """Test converting function result with error."""
+    function_result = Content.from_function_result(call_id="call_456", result="Error occurred", exception="Test error")
+
+    messages = [Message(role="tool", contents=[function_result])]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], ToolMessage)
+    assert tau2_messages[0].error is True
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_multiple_text_contents():
+    """Test converting message with multiple text contents."""
+    messages = [
+        Message(role="user", contents=[Content.from_text(text="First part"), Content.from_text(text="Second part")])
+    ]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 1
+    assert isinstance(tau2_messages[0], UserMessage)
+    assert tau2_messages[0].content == "First part Second part"
+
+
+def test_convert_agent_framework_messages_to_tau2_messages_complex_scenario():
+    """Test converting complex scenario with multiple message types."""
+    function_call = Content.from_function_call(call_id="call_789", name="complex_tool", arguments='{"key": "value"}')
+
+    function_result = Content.from_function_result(call_id="call_789", result={"output": "tool result"})
+
+    messages = [
+        Message(role="system", contents=[Content.from_text(text="System prompt")]),
+        Message(role="user", contents=[Content.from_text(text="User request")]),
+        Message(role="assistant", contents=[Content.from_text(text="I'll help you"), function_call]),
+        Message(role="tool", contents=[function_result]),
+        Message(role="assistant", contents=[Content.from_text(text="Based on the result...")]),
+    ]
+
+    tau2_messages = convert_agent_framework_messages_to_tau2_messages(messages)
+
+    assert len(tau2_messages) == 5
+    assert isinstance(tau2_messages[0], SystemMessage)
+    assert isinstance(tau2_messages[1], UserMessage)
+    assert isinstance(tau2_messages[2], AssistantMessage)
+    assert isinstance(tau2_messages[3], ToolMessage)
+    assert isinstance(tau2_messages[4], AssistantMessage)
+
+    # Check the assistant message with tool call
+    assert tau2_messages[2].tool_calls is not None
+    assert len(tau2_messages[2].tool_calls) == 1
+    assert tau2_messages[2].tool_calls[0].name == "complex_tool"