Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Waiting to run
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Waiting to run
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Waiting to run
It exposes a one-shot per-turn primitive, API-compatible with hermes-adapter
and openclaw-adapter, so the orchestrator can run the same benchmarks against
the smithers harness (--agent smithers).
How it works
Each turn spawns a one-shot bun process running smithers_adapter/smithers_turn.mjs
inside the Smithers install directory. That script drives Smithers' own
OpenAIAgent (a ToolLoopAgent built on the Vercel ai SDK) for a single turn
against an OpenAI-compatible endpoint (Cerebras gemma-4-31b by default), and
emits one JSON line: {"text", "thought", "actions", "params": {"tool_calls", "usage"}}.
Tools are declared without an execute handler, so the agent returns the
emitted tool calls for the benchmark runner to score instead of executing them
— exactly what single-turn benchmarks (BFCL, action-calling) need.
The model is forced onto the chat-completions endpoint via provider.chat(model)
because OpenAI-compatible backends such as Cerebras don't implement the newer
/responses endpoint @ai-sdk/openai defaults to.
Install
The harness needs bun on PATH and smithers-orchestrator installed. The
standard location mirrors the openclaw install convention:
mkdir -p ~/.eliza/agents/smithers/0.22.0 &&cd$_
bun add smithers-orchestrator@0.22.0 @ai-sdk/openai ai zod
Run a benchmark
cd packages/benchmarks
CEREBRAS_API_KEY=... python -m orchestrator.cli run \
--model-profile cerebras-gemma-4-31b \
--benchmarks bfcl \
--agent smithers
GEPA prompt optimization
Smithers ships GEPA-style reflective prompt optimization
(smithers optimize workflow.tsx --cases evals/*.jsonl --provider cerebras).
See docs/SMITHERS_INTEGRATION.md for how an optimized prompt artifact can be
fed back into a benchmark workflow to lift scores.
wehub-resource-sync
426e9eeabd