wehub-resource-sync
426e9eeabd
7.1 KiB
ElizaOS tau-bench Benchmark
Faithful, vendored implementation of Sierra's tau-bench (Yao et al., 2024) wired into the ElizaOS evaluation harness.
What this package contains
This package vendors the upstream tau-bench source tree under
elizaos_tau_bench/upstream/ (MIT, see elizaos_tau_bench/upstream/LICENSE)
and provides an ElizaOS-friendly runner + LLM judge + pass^k harness on top
of it.
Vendored from:
- repo:
https://github.com/sierra-research/tau-bench - commit:
59a200c6d575d595120f1cb70fea53cef0632f6b
Upstream modules used directly:
upstream/envs/base.py(Env, action/tool/reward semantics)upstream/envs/retail/(115 tools, products/users/orders DBs, wiki, rules)upstream/envs/airline/(50 tools, flights/reservations/users DBs, wiki, rules)upstream/envs/retail/tasks_test.py-- 115 official retail tasksupstream/envs/airline/tasks_test.py-- 50 official airline tasksupstream/envs/user.py-- LLM-based User simulator (LLM / React / Verify / Reflection strategies)upstream/types.py-- Action, Task, EnvResponse, ...
Total tasks loaded: 165 (115 retail + 50 airline test split). Retail also
exposes dev and train splits via --task-split.
Architecture
TauBenchRunner (elizaos_tau_bench/runner.py)
iter_tasks() -> for each task:
for trial in num_trials:
Env = upstream.get_env(...) # real DB + 115/50 tools + wiki + rules
agent.solve(Env, task_index) # multi-turn tool-calling loop:
while not done:
LLM -> Action
Env.step(Action)
if tool: invoke against real DB
if RESPOND: User.step(msg) # LLM user simulator (gpt-4o)
# produces next user turn
# until ###STOP###
judge_outputs_satisfied(...) # gpt-4o-mini gates task.outputs
pass_k.calculate_pass_hat_k(...) # unbiased pass^k estimator
The agent loop mirrors upstream ToolCallingAgent.solve: the agent emits
either a function/tool call (executed by Env) or a RESPOND action (forwarded
to the user simulator). The user simulator is a real multi-turn LLM by
default (--user-model gpt-4o); the rollout ends when it emits
###STOP### or agent_max_turns is reached.
Installation
cd packages/benchmarks/tau-bench
pip install -e ".[dev]"
Core dependencies: httpx, pydantic, pytest, pytest-asyncio.
litellm is optional (pip install -e ".[litellm,dev]"). When it is not
installed, the built-in agent, user simulator, and judge can call an
OpenAI-compatible chat-completions endpoint by setting
TAU_BENCH_OPENAI_BASE_URL or OPENAI_BASE_URL (for example a local
llama.cpp server at http://127.0.0.1:8080/v1).
Required environment variables
| Component | Env var (default provider = openai) | Default model |
|---|---|---|
| Agent LLM | OPENAI_API_KEY |
gpt-4o |
| User simulator | OPENAI_API_KEY |
gpt-4o |
| LLM judge | OPENAI_API_KEY |
gpt-4o-mini |
Override providers with --agent-provider, --user-provider, --judge-provider.
With LiteLLM installed, any LiteLLM provider works. Without LiteLLM, use
openai-compatible, local, or llama.cpp with an OpenAI-compatible base URL.
The runner checks each provider's API key up front and refuses to start unless
every required one is present (or you pass --mock).
Upstream data assets
The repository ships only compact smoke fixtures for the sample task IDs. Full
retail and airline JSON data is fetched lazily from sierra-research/tau-bench
into ~/.cache/elizaos_tau_bench/upstream/<ref>/... when an official benchmark
run first needs it. Set TAU_BENCH_DATA_DIR to use a pre-populated local copy,
TAU_BENCH_DATA_MODE=smoke for fixture-only sample runs, or
TAU_BENCH_DISABLE_DATA_DOWNLOAD=1 to require local files and fail if missing.
Quick start
Run the full 165-task benchmark with pass^4 (paper default):
python -m elizaos_tau_bench --num-trials 4
Run only retail dev split with a single trial:
python -m elizaos_tau_bench --domain retail --task-split dev --num-trials 1
Run a tiny smoke (4 sample tasks) with the deterministic mock agent -- no LLM, no API keys required:
python -m elizaos_tau_bench --mock --use-sample-tasks
Run a specific task subset:
python -m elizaos_tau_bench --domain retail --task-ids 0 1 2 --num-trials 4
Use a non-OpenAI model for the agent but keep openai for user/judge:
python -m elizaos_tau_bench \
--agent-provider anthropic --agent-model claude-3-5-sonnet-latest \
--user-provider openai --user-model gpt-4o \
--judge-provider openai --judge-model gpt-4o-mini
Disable the LLM judge (fall back to upstream's literal substring check):
python -m elizaos_tau_bench --no-llm-judge
Pass^k
pass^k = probability that all k independent trials of a task succeed. We compute the unbiased estimator from the paper:
pass^k = E_task [ C(c, k) / C(n, k) ]
where n = total trials, c = successful trials, per task. Defaults to k in {1,2,4}
and --num-trials 4 (paper default).
LLM judge
Upstream's reward gate for task.outputs is a literal substring search in the
agent's RESPOND messages -- brittle when the agent paraphrases correctly. The
judge here calls a small LLM (--judge-model gpt-4o-mini) with a strict JSON
schema:
{"per_output": {"<output>": true|false, ...}, "explanation": "..."}
A task passes when both (a) the upstream data-hash check succeeds (agent's actions reach the same DB state as the ground-truth actions) and (b) every required output is satisfied per the judge. If the judge LLM is unavailable (missing API key, transient failure, unparseable response) it falls back to the upstream substring check and records that in the explanation.
Programmatic API
from elizaos_tau_bench import TauBenchConfig, TauBenchRunner
cfg = TauBenchConfig(
domains=["retail", "airline"],
task_split="test",
num_trials=4,
agent_model="gpt-4o",
user_model="gpt-4o",
judge_model="gpt-4o-mini",
output_dir="./out",
)
report = TauBenchRunner(cfg).run()
print(report.pass_k[4].pass_hat_k)
report.results is the per-(task, trial) list; report.to_dict() serializes
everything in a stable JSON shape that's also written to
<output_dir>/report.json and <output_dir>/trajectories.json.
Tests
pytest packages/benchmarks/tau-bench/
Tests cover dataset loading, pass^k math, judge behaviour, completion adapter fallbacks, output JSON contracts, and an end-to-end smoke that runs one retail task with a stubbed completion adapter -- verifying the multi-turn user-simulator loop runs and at least one tool call lands.
License
This package is MIT-licensed. The vendored elizaos_tau_bench/upstream/ tree
is (c) 2024 Sierra and distributed under the upstream MIT license -- see
elizaos_tau_bench/upstream/LICENSE. Copyright headers (# Copyright Sierra)
are preserved verbatim across every vendored file.