vercel-labs--zerolang/evals

Zero Evals

TypeScript evals for agent-facing Zero workflows.

Run the checked-in fixture without calling Claude:

pnpm evals -- --case hello-world --fixture
pnpm evals -- --case rosetta-100-doors --fixture
pnpm evals -- --suite agent-scale --fixture

Run live in Vercel Sandbox through Claude Code and Vercel AI Gateway:

AI_GATEWAY_API_KEY=... pnpm evals -- --case hello-world

By default, evals run each selected case against:

• anthropic/claude-opus-4.7
• anthropic/claude-sonnet-4.6

Override the model set with repeated --model flags or a comma-separated --models value:

pnpm evals -- --case hello-world --model anthropic/claude-sonnet-4.6
pnpm evals -- --case hello-world --models anthropic/claude-opus-4.7,anthropic/claude-sonnet-4.6

Live evals create a Vercel Sandbox, upload the current checkout, build the native compiler, install Claude Code, and run the agent inside the sandbox. Each model/case run gets a fresh copy of the prepared checkout so mutations from one run do not affect the next run. The sandbox network policy injects the AI Gateway bearer credential for https://ai-gateway.vercel.sh.

Credential options:

# Sandbox auth
VERCEL_OIDC_TOKEN=...
# or VERCEL_TOKEN=... VERCEL_TEAM_ID=... VERCEL_PROJECT_ID=...

# AI Gateway auth
AI_GATEWAY_API_KEY=...

# Model selection
ZERO_EVAL_MODELS=anthropic/claude-opus-4.7,anthropic/claude-sonnet-4.6
# or ZERO_EVAL_MODEL=anthropic/claude-sonnet-4.6

Each live model run must load Zero's version-matched skill through bin/zero skills get zero --full, then use bin/zero check and bin/zero run inside the sandbox to verify its candidate.

The Rosetta cases are deterministic code-challenge evals. Prompts describe the task behavior and expected output; the evaluator does not compare an exact projection. It imports the returned source into a graph artifact, checks that graph, runs it, and compares stdout/stderr plus a small set of source-shape requirements.

The agent-scale suite covers larger agent tasks. It includes multi-command CLI programs with several runtime checks and graph package fixtures such as a CRM HTTP request-envelope API. Package cases are validated in place: the evaluator runs zero check, executes each smoke route or command with isolated --out paths, and inspects zero view output for required graph/source shape signals.

The eval system prompt intentionally avoids Zero syntax examples. The model is expected to learn task-relevant syntax from the version-matched skills and compiler feedback.