An eval harness for AI-agent authoring of Native SDK apps. It formalizes the "clean-agent trial": give a fresh agent nothing but a scaffolded workspace, the native-ui skill, and a task prompt, then grade what it produced deterministically.

Per case the runner:

Scaffolds a fresh workspace with the repo's own CLI — zig build at the repo root, then zig-out/bin/native init evals/.workspaces/<case> --frontend native (--template zig-core for the pre-existing native cases and the zig side of dual cases, --template ts-core for the ts side) — and delivers each track's skills exactly the way a real user gets them: native skills get <name> written to .claude/skills/<name>/SKILL.md (init does not ship skills). The zig track gets native-ui + zig; the ts app track gets ts-core + native-ui; core-only ts-core cases get ts-core. The workspace is then pre-warmed (native test once — workspaces are zero-config, so builds go through the CLI verbs) so the agent's own builds are incremental and its wall-clock isn't spent compiling the SDK.
Runs the agent-under-test: claude -p "<task prompt>" headless in the workspace, routed through the Vercel AI Gateway, with a per-run CLAUDE_CONFIG_DIR so no user-level memory/plugins/hooks leak in, --max-turns, a wall-clock timeout, and the full stream-json transcript captured to results/.
Grades with deterministic checks: native test in the workspace, native markup check on the .native files, per-case file greps (e.g. "the board uses <template>"), and live automation-snapshot greps (native build with -Dautomation=true, launch, wait for the automation snapshot, grep it for expected roles/names).
Judges quality the deterministic checks can't see — idiomatic Model/Msg design, template factoring, test meaningfulness — with an llm_judge check: a judge model called directly through the gateway scores case-specific criteria 0–10 against the task prompt and the agent's code. Advisory by default (the score is recorded and printed but never fails the case); set "advisory": false on a case to make minScore a gate. Skipped in --dry-run.
Reports a per-case result.json (pass/fail per check, judge scores, durations, model, turns, cost) plus a console summary table.

Requirements

macOS (live snapshot checks launch the app; use --skip-live elsewhere, or --sandbox for the Linux lane), Zig 0.16.0, node >= 24, pnpm 10.x.
The Claude Code CLI (claude) on PATH.
A Vercel AI Gateway API key for real runs:

export AI_GATEWAY_API_KEY="vck_..."   # or VERCEL_AI_GATEWAY_API_KEY

The runner assembles the gateway env for the claude subprocess per Vercel's Claude Code guide:

ANTHROPIC_BASE_URL=https://ai-gateway.vercel.sh
ANTHROPIC_AUTH_TOKEN=$AI_GATEWAY_API_KEY
ANTHROPIC_API_KEY=            # empty string on purpose: a non-empty value would win over the auth token

Models are gateway slugs. The coder (agent-under-test) defaults to anthropic/claude-sonnet-5 (override with --model or NATIVE_SDK_EVAL_MODEL); the judge defaults to anthropic/claude-opus-4.8 (override with --judge-model or NATIVE_SDK_EVAL_JUDGE_MODEL). Both tracks of a dual case always run with the SAME coder model — fairness is a property of the run, so set it once at run time.

Usage

cd evals
pnpm install

pnpm eval --list                      # list cases
pnpm eval --dry-run                   # everything except the model call (no key needed):
                                      #   scaffold + skill delivery, print env + claude argv,
                                      #   run the graders against the untouched scaffold
                                      #   (grader FAILs are expected there and exit 0)
pnpm eval templates-settings-app      # one real run
pnpm eval                             # the whole suite
pnpm eval --model anthropic/claude-opus-4.8 templates-settings-app
pnpm eval --judge-model anthropic/claude-fable-5 templates-settings-app
pnpm eval --skip-live                 # skip snapshot checks (no app launch / non-macOS)
pnpm eval --track ts dual-feed-table  # one track of a dual-track case (default: both)
pnpm eval --keep-workspaces           # keep .workspaces/<case> around for inspection
pnpm eval --trials 5 expenses-table   # 5 independent trials per case; report pass rates
pnpm eval --concurrency 3             # run up to 3 case trials in parallel (default 2 locally)
pnpm eval --sandbox                   # run each case in its own Vercel Sandbox microVM
pnpm eval --sandbox --dry-run         # full sandbox path minus the model call
pnpm eval --sandbox --sandbox-vcpus 8 # bigger sandboxes (2048 MB RAM per vCPU)
pnpm typecheck

Cases run in parallel (log lines are prefixed [case-name]); --concurrency caps how many at once — locally the default is 2 to keep zig builds from thrashing, with --sandbox the default is 4 (each case has its own VM, but plans rate-limit vCPU allocation).

Trials

Model runs are stochastic; a single pass or fail is weak evidence. --trials <n> runs each case n times, each trial fully independent — its own scaffolded workspace (.workspaces/<case>-trial-<n>), its own agent run, its own checks and judge call — and reports per-case pass rates (e.g. 3/5), per-check pass counts, and the mean judge score. Trials share the --concurrency pool (log lines are prefixed [case-name#trial]), and with --sandbox each trial gets its own microVM.

With --trials 1 (the default) the behavior and file layout are exactly the single-run layout described above. With --trials > 1 each case directory nests per-trial results plus an aggregate:

results/<stamp>/
  summary.json                      # array of per-case aggregates
  <case>/
    aggregate.json                  # pass rate, per-check pass counts, mean judge score, per-trial results
    trial-1/result.json             # exactly a single-run result.json (plus a "trial" field)
    trial-1/transcript.jsonl
    trial-2/...

The summary table swaps the PASS/FAIL column for a pass rate column, the checks column shows per-check pass counts (3/3 2/3 ..., s = skipped in every trial), judge is the mean score, cost/time are totals across trials, and a per-check breakdown is printed under the table. A real run exits non-zero if any trial failed.

Lanes

Every result carries a lane — where the case ran and got graded — and the summary table has a lane column:

macos-local (default): the run described above; live snapshot checks launch the app directly and require macOS.
linux-sandbox (--sandbox): each case trial runs in its own isolated Vercel Sandbox microVM booted from a pre-baked Linux image, including the live checks — the app builds with -Dplatform=linux -Dweb-engine=system -Dautomation=true, launches under Xvfb, and is driven through the same automation dropbox the macOS lane uses. An engine screenshot of the app's gpu_surface is captured through the dropbox and pulled back with the results.

A check that greps a surface which exists on only one OS can declare "lanes": ["macos-local"] in eval.json; on other lanes it reports skipped (not graded on the ... lane) instead of failing, so the summary distinguishes "fails" from "not applicable on this lane". Audited 2026-07-05: none of the ten shipped cases needs a lane annotation — every snapshot pattern asserts roles/names from the SDK's own automation snapshot, and the surfaces they cover (secondary windows, gpu charts, trees, tables) are proven on Linux by tools/linux-truth/. A Linux-lane failure is therefore a real failure until a case says otherwise.

Vercel Sandbox mode

--sandbox boots each case trial from a custom image (see evals/sandbox/) that bakes the Linux GUI stack (GTK4 + WebKitGTK + Xvfb), zig, node/pnpm, the Claude Code CLI, and a pre-warmed build layer: the repo at a pinned ref with the CLI, workspace-test, and automation build graphs already compiled into fixed cache paths. Per case the runner then:

uploads the repo working tree as a tarball and rsyncs it over the baked repo — deletions propagate, caches survive, so builds against the current tip are incremental on top of the bake;
starts Xvfb and re-invokes this same harness inside (pnpm eval --skip-permissions --lane linux-sandbox <case>) with the gateway env assembled exactly like a local run;
pulls the whole case results directory home — result.json, transcript.jsonl, live-*.png engine screenshots — before the microVM is destroyed.

--dangerously-skip-permissions is safe there because the whole VM is the throwaway. --sandbox --dry-run exercises the entire path — provisioning, refresh, scaffold, graders (which FAIL against the untouched scaffold, as designed), artifact pull — without the model call, and exits 0.

Image: build and push once with evals/sandbox/build-image.sh (needs a Docker login to the registry — the two token variants are in the script header). Rebuild when the Dockerfile changes, when zig bumps, or when the tip has drifted far enough from the baked ref that in-sandbox builds stop feeling incremental; runs stay correct without a rebuild because of the working-tree refresh. After a push the registry prepares the image for a few minutes; the runner retries image_not_ready for up to 10 minutes. --sandbox-image overrides the default reference (eval-sandbox, resolved in the linked project).

Auth (checked before any sandbox work): either an OIDC token — one-time setup in evals/:

vercel link --scope vercel-labs --project zero-native
vercel env pull .env.local   # the runner auto-loads VERCEL_OIDC_TOKEN from .env.local

(the token expires ~12h; re-run vercel env pull .env.local when sandbox auth fails) — or VERCEL_TOKEN + VERCEL_TEAM_ID + VERCEL_PROJECT_ID for environments where OIDC is unavailable. Real runs additionally need AI_GATEWAY_API_KEY as usual.

Cost and limits: sandbox compute is metered (active CPU + provisioned memory) — a 4-vCPU case trial that runs 20-30 minutes lands around $0.20-0.35 plus the model tokens through the gateway. Sandboxes cap at 45 minutes wall clock on the Hobby plan (the runner's per-sandbox timeout), and vCPU allocation is rate-limited per plan, which is why the default --concurrency in sandbox mode is 4.

Real runs exit non-zero if any case fails. Workspaces live in .workspaces/ and results in results/<timestamp>/<case>/ (result.json, transcript.jsonl, the isolated claude-config/ — kept in-VM and not pulled for sandbox runs, plus live-*.png screenshots from the Linux lane); both directories are gitignored.

Permissions for the agent-under-test

By default the agent runs with --permission-mode acceptEdits plus an allowlist covering zig ..., native-sdk ..., and basic file commands — enough for unattended edit/build/test loops without granting arbitrary shell. --skip-permissions switches to --dangerously-skip-permissions; only use it if the default allowlist blocks a case, and remember the workspace is a throwaway dir but the process is not otherwise sandboxed.

In both modes the runner passes --disallowedTools deny rules for evals/cases/** and evals/results/**: the workspace references the SDK repo by path, so the harness itself is reachable from the agent's cwd, and agents exploring the repo for docs/examples were observed reading their own grading config (3/20 runs of the 2026-07-04 suite). SDK source and examples stay readable on purpose — a real user has the repo.

Cases

templates-settings-app — validates the new grammar: repeated grouped toggle sections where <template>/<use> is the natural shape, plus token style attributes (muted headers, surface cards). Checks: build+tests, markup check, <template>/<use> greps, token-attribute greps, snapshot roles.
kanban-board — port of the manual builder trial; card identity must survive moving between columns (global-key).
habits-tracker — port of the manual markup trial; text entry (elm-style mirror), derived/filtered lists, enum filters.
expenses-table — exercises the newest grammar (every built-in component markup-expressible): an expense ledger whose natural shape is table > table-row > table-cell with <for> rows, an exclusive category filter, and an alert-shaped empty state. The prompt describes only requirements (rows-and-columns of data, a callout, pinned display strings); the greps assert the agent reached the table grammar from the skill alone.
process-monitor — exercises the effects surface: a long-running local command (a harmless sh -c tick loop the prompt pins exactly) spawned from update through the effects channel, lines streaming into a bounded 12-line list, cancel by model-owned key, and status/counts derived from the line/exit Msgs. Greps assert .update_fx wiring, fx.spawn/fx.cancel, fake-executor tests, and the absence of hand-rolled std.process.Child/std.Thread; the live snapshot asserts the idle state (Start/Cancel, "Status: idle", "0 lines · 0 dropped") so nothing spawns during grading. The judge scores effect-key discipline, non-blocking behavior, honest drop accounting, and derive-don't-store.
release-dashboard — exercises the pipeline composites and markdown-in-markup: a release dashboard whose natural shape is <stepper> for the five-stage track (starting on "Canary"), <timeline>/<timeline-item> for the seeded event history, and a <markdown>-sourced notes panel containing a GFM table. Greps assert all three elements plus <for> events; the snapshot asserts the composite semantics ("Canary (active)" stepper labels, timeline items by title, role=gridcell markdown table cells, the pinned status summary). The judge scores house-style composition (no ad-hoc reimplementations) and declarative markup.
settings-picker — exercises the anchored floating surface pattern: two picker rows (Theme/Accent) whose natural shape is a <select> trigger + <if>-mounted <dropdown-menu anchor=...> of <menu-item>s with on-dismiss, model-owned open state, and an exclusive-open rule. Greps assert the select/anchored-dropdown/on-dismiss/menu-item composition; the snapshot asserts the closed idle state and the pinned status line (the trigger's accessible name may be its value text or an explicit label — both legit, the pattern allows either).
file-browser-panes — exercises split panes and the tree keymap: a resizable two-pane browser (<split value= on-resize=> with pane min-width floors) whose sidebar is a <tree> of role="treeitem" rows with model-owned expanded state driving a detail pane. The snapshot asserts role=separator/role=tree/role=treeitem, the seeded rows, and the pinned selection status. The judge is explicitly told the tree keymap is engine-provided so it never expects app-side keyboard Msgs.
metrics-dashboard — exercises the markup chart element and icon-in-button: the whole view is markup (header with <button icon=...>, the <chart> with a <series values="{binding}">, status bar) and the Zig side owns only model/update logic. Greps assert <chart, icon=", and the pinned y-min="0"; the snapshot asserts role=chart plus the seeded summary.
inspector-window — exercises model-declared secondary windows: a task list whose inspector opens as a real second OS window through Options.windows_fn + window_view, live-updates with the selection, reflects a user close via on_close, and never duplicates. A negative grep rejects faking it with <dialog>/<sheet>/<drawer>.
disk-status-spans — exercises inline text spans: one wrapped paragraph mixing bold, muted, and monospace runs (announcing as a single text run), with bound values interpolated inside the spans and a derived percentage summary. Greps assert the three span styles inside one enclosing <text>; the snapshot asserts the paragraph as ONE combined text run with resolved bindings.
chat-composer — exercises the composer composites: an <input-group> wrapping a <textarea> plus an in-border <input-group-actions> row, an attached <button-group> cluster, a derived disabled Send, and a <for> message list. The snapshot asserts the composer textbox, the attached cluster, Send disabled on the empty draft, the seeded messages, and the live count.
brand-icon-package — exercises the one-image app-icon pipeline through native package: the manifest sources icons from a single square PNG at the brand path, no prebuilt icon containers anywhere, and the packaged macOS bundle carries a generated AppIcon.icns wired into its metadata.
playlist-row-actions — exercises the interaction seams: row-level Enter as a list row's primary action (on-submit on list-item, distinct from select-on-press) and the app-level key fallback (Options.on_key) for Space and the arrows when nothing is focused. Greps assert the row-level submit binding and the fallback wiring; the snapshot asserts the seeded rows, the model-driven selection, and the idle derived status.

Add a case by creating cases/<name>/eval.json (see src/types.ts for the schema). Prompts describe app requirements, never the solution — the point is to see whether a fresh agent reaches the intended grammar from the skill alone.

The TS-authoring track

Cases with "frontend": "ts-core" measure the other authoring surface: the app core written in the TypeScript subset and compiled by packages/core. The scaffold is not an app — it is src/core.ts (the case's starter/ overlay when it ships one, else a minimal counter core), a README with the check loop, and the ts-core skill delivered via native skills get ts-core. Two graders replace build/markup/snapshot:

ts_transpile — the transpiler must exit clean on src/core.ts: tsc-semantics typecheck, every subset rule (NS1001-NS1050), Zig emission. Failing diagnostics stay in the result as the violation evidence.
ts_harness — behavioral grading: transpile the core, assemble a scratch dir with the emitted core.zig, the rt kernel, and the case's harness.zig, then zig test harness.zig. The harness drives the real dispatch cycle (update → commitModelRoot → frameReset) and asserts the prompt's requirements, so the case prompt pins the Model/Msg/export contract exactly (an API spec, not a solution).

Because the grading harness compiles against the agent's code, ts-core prompts pin names; behavior stays requirements-only. The transpiler compiles pure update(model, msg): Model cores and effectful Model | [Model, Cmd<Msg>] pair-returns (the Cmd surface); the wave-2 dual-track cases below are the effects coverage.

The ts-core cases:

ts-habits-core — a fresh core from a requirements description (the TS mirror of habits-tracker): seeded habits, toggle-with-streak math, a trimmed draft add path, exclusive filters, derived counts.
ts-expenses-filter — a feature-add to an existing subset file: extend a working expense ledger (shipped as starter/) with an exclusive Category | null filter and derived filtered views, without breaking the existing behavior.
ts-countdown-bugfix — a bug-fix in an existing subset file: a countdown core whose tick/reset/set_duration transitions drifted from the spec (unguarded ticks, no completion stop/floor, hardcoded reset); the prompt states the intended behavior, the harness isolates each drift.
ts-tag-input-core — the text seam under pressure: comma-separated tag parsing with byte-level trim, empty-drop, and case-insensitive dedupe, where string methods are the reflex and NS1004 says no.

The dual-track cases (wave 2)

Wave 1's four ts-* cases measured subset compliance on toy katas and compared against Zig numbers gathered before the zig 0.16-idioms skill existed. Wave 2 replaces that comparison with an honest, contemporaneous one: six realistic asks, each ONE language-blind spec ("frontend": "app-dual") that runs on both authoring tracks — <case>@ts scaffolds a full TypeScript app (native init --frontend native --template ts-core), <case>@zig the Zig app template (--template zig-core). Identical prompt, identical shared checks, one behavioral spec asserted by two thin per-track harnesses; --track ts|zig selects a lane, the default runs both.

Grading per track: the shared checks (native test -Dplatform=null, markup check, view greps, the judge with a case rubric) plus the track's behavioral harness. On the ts track, ts_harness transpiles src/core.ts (its whole import graph) and zig tests the case's harness-ts.zig against the emitted core, the rt kernel, and harness-lib/cmdview.zig — a decoder over the Cmd/Sub wire format, so harnesses assert effects semantically ("one GET to the pinned URL", "the delay re-armed on the same key"). On the zig track, zig_harness injects the case's harness-zig.zig into the workspace as src/eval_behavior_spec.zig (a test import appended to src/main.zig, restored afterward) and runs native test, so it compiles against the agent's real Model/Msg/update and drives the SDK's deterministic fake effects executor (fx.executor = .fake, pendingSpawnAt/feedLine/feedExit/fireTimer/feedResponse/feedFileResult).

The cases — every prompt reads like a real user ask, and every effect result is fed by the harness (no network, no processes, no clocks during grading):

dual-feed-table — "fetch JSON from this API and show it in a sortable table": the HTTP effect (pinned URL, buffered GET, keyed), JSON parsing of a pinned flat schema, two sort orders derived at view time, and the five honest states (idle / loading with re-entry guarded / loaded / explicit empty / failed keeping previous rows) across non-200, transport-failure, and malformed-body deliveries.
dual-notes-autosave — "add debounced autosave to this notes app" (starter provided per track, both riding the SDK text-input engines): the keyed one-shot re-arm debounce (Cmd.delay / fx.startTimer replace), byte-exact notes.tsv serialization, Save now with the pending autosave cancelled, a five-state save lifecycle, and the late-result race (a save landing after newer edits must not mark them saved).
dual-pomodoro-timer — "build a pomodoro timer with a completion sound": the recurring timer surface (Sub.timer declared from the model / fx timer armed-cancelled through the channel — armed exactly while running), auto-advancing focus/rest state machine, work-only completion counting, stale-tick guards, and the audio surface at completions.
dual-list-delete-fix — "this list doesn't update after delete — fix it": a realistic seeded bug (a hand-maintained visible-list cache that delete forgot) in a medium app with a markup view; the harness pins list correctness across delete/toggle/add/filter, the judge scores whether the fix is the derive-don't-store root cause or a symptom patch.
dual-ledger-csv-split — "split this core into modules and add export-to-CSV in the new module": multi-file authoring (relative imports on ts, @import on zig, both grepped), byte-exact CSV quoting (commas and doubled quotes exercised by the seeds), the file-write effect, and a four-state export lifecycle.
dual-sysinfo-panel — "show system info from a shell command in the UI": a collect-mode background spawn of a pinned argv, byte parsing of the output line, re-entry guarded while probing, and honest ok / failed-with-code / never-started states.

Starters (starter-ts/, starter-zig/) overlay the scaffold for the feature-add, bug-fix, and split cases — written in each track's idiom (immutable spread cores over the SDK text engine on ts; bounded buffers, canvas.TextBuffer, and Model-method bindings on zig) so neither track starts from translated code.

Fairness and methodology. Both tracks get identical prompts (language-neutral asks; the pinned contract names arms/fields in the wire spelling and states each track's idiom for the few surfaces that necessarily differ — routed err arms on ts vs payload-carried outcomes on zig). Each track gets its CURRENT skill set by the documented delivery path and nothing extra: ts → ts-core + native-ui; zig → native-ui + zig (the 0.16-idioms skill). The same coder model grades both tracks of a run. Known caveats to carry into any writeup: wave-1's Zig numbers predate the zig skill (this suite is the corrected baseline — the pre-existing native cases now also receive it, so their history splits at this commit); the behavioral harnesses are compiled contracts, so a run can fail for contract-shape reasons the prompt pins explicitly; and the two tracks' harness glue differs by necessity (wire-format decoding vs fake-executor introspection) while asserting the same spec, test for test.

Authoring metrics

pnpm metrics results/<stamp> [...] post-processes finished runs' transcripts into the agent-authoring metrics the checks cannot see, per case and per track — ts (the ts-core cases and the @ts side of dual cases) vs zig (the pre-existing native cases and the @zig side): first-pass compliance (did the agent's first compliance check after touching sources pass — the transpiler run, native check, native test, or native build on the ts track; native test / native build / native check / native markup check on the zig track), retries-to-green (failing compliance runs before the first green), teaching-error encounters (failing compliance runs that carried a teaching diagnostic — the "did the diagnostics work" round-trip count wave 2 compares across tracks), violation taxonomy (NS/TS rule IDs on the ts track; zig error lines on the zig track, with no member named 'X' bucketed by member so the 0.16-idiom class is visible) raw and per 1k generated LOC (lines written through Write/Edit to source files, .native markup included), and task success (the run's own pass verdict). Harness friction — permission-refused commands, errored compounds with no diagnostic in the output — is dropped from the event stream so it never masquerades as an authoring failure. It writes authoring-metrics.json next to each summary.json.

CI

CI only typechecks the harness (evals-typecheck in .github/workflows/ci.yml). Eval runs are local/manual — no model calls in CI.

README.md Unescape Escape

Native SDK evals