elizaos--eliza/plugins/plugin-benchmarks/AGENTS.md

@elizaos/plugin-benchmarks

Canonical elizaOS Action wrappers for benchmark tool vocabularies (vending-bench, webshop, OSWorld, tau-bench, visualwebbench).

Purpose / role

This plugin adds a fixed, stable set of elizaOS actions that mirror the tool vocabularies of five standard agent evaluation benchmarks. Each action captures benchmark-specific parameters in a structured, typed form so that fine-tuning on benchmark traces produces consistent action names regardless of which benchmark the trace came from. The plugin is opt-in — include benchmarksPlugin in the plugins array of an AgentRuntime to enable it.

Plugin surface

The exported plugin object benchmarksPlugin registers 37 actions total — five umbrella actions promoted into per-subaction virtuals via promoteSubactionsToActions from @elizaos/core:

Umbrella action	Subactions (promoted as <UMBRELLA>_<SUBACTION>)	Count
VENDING_MACHINE	view_state, view_suppliers, place_order, restock_slot, set_price, collect_cash, update_notes, check_deliveries, advance_day	1 + 9
WEBSHOP	search, click, select_option, back, buy	1 + 5
OSWORLD	click, double_click, right_click, type, key, scroll, drag, screenshot, wait, done, fail	1 + 11
TAU_BENCH_TOOL	None (tool_name is free-text, not an enum; no virtuals promoted)	1
VISUALWEBBENCH_TASK	web_caption, webqa, heading_ocr, element_ocr, element_ground, action_prediction, action_ground	1 + 7

No providers, services, evaluators, routes, or events are registered.

Layout

plugins/plugin-benchmarks/
  index.ts                     Re-exports from src/index
  src/
    index.ts                   Plugin assembly — imports actions, calls promoteSubactionsToActions, exports benchmarksPlugin
    actions/
      vending-machine.ts       VENDING_MACHINE action (vending-bench)
      webshop.ts               WEBSHOP action (WebShop benchmark)
      osworld.ts               OSWORLD action (OSWorld desktop-control benchmark)
      tau-bench.ts             TAU_BENCH_TOOL action (tau-bench retail/airline tools)
      visualwebbench.ts        VISUALWEBBENCH_TASK action (VisualWebBench vision tasks)
  __tests__/
    plugin.test.ts             Vitest suite — verifies action counts, umbrella names, promoted virtuals
  build.ts                     Build script (Bun.build)
  vitest.config.ts             Test config
  package.json

Commands

All scripts are defined in package.json and scoped to this package:

bun run --cwd plugins/plugin-benchmarks build       # compile to dist/
bun run --cwd plugins/plugin-benchmarks dev         # hot-rebuild during development
bun run --cwd plugins/plugin-benchmarks test        # run vitest suite
bun run --cwd plugins/plugin-benchmarks typecheck   # tsgo --noEmit
bun run --cwd plugins/plugin-benchmarks lint        # biome check --write --unsafe
bun run --cwd plugins/plugin-benchmarks lint:check  # biome check (read-only)
bun run --cwd plugins/plugin-benchmarks format      # biome format --write
bun run --cwd plugins/plugin-benchmarks clean       # rm -rf dist .turbo

Config / env vars

None. This plugin reads no environment variables and has no runtime configuration. It is purely a vocabulary shim — real benchmark execution happens in the benchmark harness environment, not inside the action handlers.

How to extend

Add a new benchmark action

1. Create src/actions/<bench-name>.ts. Export a named Action constant.
   • If the benchmark has a fixed set of operations, use a const enum array for the action parameter and call promoteSubactionsToActions in src/index.ts.
   • If operations are dynamic (like tau-bench), register the action directly without promotion.
2. Add the import and export to src/index.ts.
3. Add the action (or its promoted virtuals) to the actions array in benchmarksPlugin.
4. Add a test case in __tests__/plugin.test.ts covering the umbrella name, promoted virtual names, and expected action count.

Add a subaction to an existing benchmark

1. Add the subaction name to the const array in the relevant src/actions/*.ts file (e.g., VENDING_SUBACTIONS).
2. Update the parameters[0].schema.enum accordingly — the array spread keeps them in sync automatically.
3. Update the expected count in __tests__/plugin.test.ts.

Conventions / gotchas

• Handlers are pass-through adapters. Every action handler returns success: true with the structured parameters passed through as data. The actual benchmark operation is performed by the external benchmark environment, not by the handler. Do not add real side effects here.
• promoteSubactionsToActions from @elizaos/core reads the first parameter's schema.enum array and generates one virtual Action per entry, named <UMBRELLA>_<SUBACTION_UPPERCASED>. The umbrella action itself is also included. Tau-bench uses the umbrella action directly because tool_name has no enum.
• subActions is mutated by promoteSubactionsToActions. After calling it, the umbrella action's subActions array is populated (9 for vending, 5 for webshop, 11 for OSWorld, 7 for visualwebbench). Tests assert these counts.
• No external dependencies beyond @elizaos/core. No native addons, no network calls, no file I/O.
• Total action count is 37. Tests assert this — update the assertion whenever actions are added or removed.
• See the root AGENTS.md for repo-wide architecture rules, logger conventions, ESM constraints, and naming standards.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — eval / trajectory harness:

• A live-model scenario run producing the JSON report + run viewer + native jsonl, with the trajectory opened and reviewed.
• The harness's own e2e tests against a real AgentRuntime — not a mocked runtime; assert outcomes, not routing (see #9970).
• Determinism/seed handling and the failure/partial-run reporting paths.
• The shape of the corpus/records emitted, inspected by hand.