wehub-resource-sync
e30e75b5d4
Changesets / Create Version PR (push) Has been cancelled
Deploy Shadcn Registry / Deploy Production (push) Has been cancelled
Template Metrics / LOC + Bundle Size (push) Has been cancelled
Code Quality / Oxlint + Oxfmt (push) Has been cancelled
Code Quality / Template Sync (push) Has been cancelled
Code Quality / Build Changed Packages (push) Has been cancelled
Code Quality / Test Changed Packages (push) Has been cancelled
Deploy Expo Example / Deploy Production (push) Has been cancelled
Deploy Ink Example / Deploy Production (push) Has been cancelled
Python Tests / pytest (assistant-stream, 3.10) (push) Has been cancelled
Python Tests / pytest (assistant-stream, 3.12) (push) Has been cancelled
Python Tests / pytest (assistant-ui-sync-server-api, 3.10) (push) Has been cancelled
Python Tests / pytest (assistant-ui-sync-server-api, 3.12) (push) Has been cancelled
102 lines
4.7 KiB
Markdown
102 lines
4.7 KiB
Markdown
# Prompt evals
|
||
|
||
A tiny A/B harness for one question: **does this guidance sentence actually
|
||
change behavior?** `AGENTS.md` is loaded into every agent, so every line there
|
||
has a cost. A sentence earns its place only if it measurably fixes a mistake an
|
||
undirected agent makes — otherwise it's noise.
|
||
|
||
## How it works
|
||
|
||
Each **case** seeds an isolated sandbox with files, hands the agent a realistic
|
||
task (e.g. "apply this PR-review feedback"), and judges the result against a
|
||
rubric. We run every **candidate** guidance string — including an empty
|
||
`baseline` — and compare pass rates:
|
||
|
||
- `baseline` should **reproduce the mistake** (low pass rate). If it doesn't,
|
||
the case isn't testing anything.
|
||
- A candidate **earns its place** if it lifts the pass rate to ~100%.
|
||
- Among candidates that work, the **shortest** wins. That's the line we add.
|
||
|
||
The agent runs via the `claude` CLI in a throwaway `/tmp` sandbox (so it sees no
|
||
`AGENTS.md` except the guidance we inject through `--append-system-prompt`). A
|
||
fresh `claude` instance acts as the LLM judge.
|
||
|
||
## Findings: comment hygiene
|
||
|
||
The `pr-review-comments` case seeds a config field that already carries a
|
||
change-narration comment (`// bumped from 5000 to 8000 …`) and asks the agent to
|
||
bump the value again. An undirected agent reliably keeps narrating the history
|
||
instead of deleting a comment that only ever described a past change.
|
||
|
||
Pass rate by guidance, on both a small and a frontier agent model (judge:
|
||
Sonnet 4.6):
|
||
|
||
| candidate | guidance injected | Haiku 4.5 | Opus 4.8 |
|
||
| --- | --- | ---: | ---: |
|
||
| baseline | _(none)_ | 0–13% | 0% |
|
||
| describe-now | "Comments describe the code as it is, not how it changed." | 0% | 0% |
|
||
| why-not-what | "Comments explain why the code is the way it is; they never narrate what changed." | 13% | — |
|
||
| no-history | "Never write comments that reference the PR, the review, or a previous version of the code." | 25% | — |
|
||
| drop-tombstones | "Code comments describe the current code, never its history. When you edit a line, remove any nearby comment that just narrates a past change." | 75% | 67% |
|
||
| **delete-stale** | **"When you change code, delete any comment that only records its history."** | 50% | **~94%** |
|
||
|
||
(Haiku at n=8; Opus `baseline`/`delete-stale` confirmed at n=6 then n=10 →
|
||
0/16 and 15/16.)
|
||
|
||
Three things fell out of this:
|
||
|
||
1. **Telling the model how to _write_ comments doesn't make it _remove_ a stale
|
||
one.** The "write good comments" phrasings (`describe-now`, `why-not-what`,
|
||
`no-history`) sit in the noise around baseline on both models — the agent
|
||
reads them as advice for new comments, not a mandate to clean up the
|
||
existing one. Only guidance that explicitly says to _delete_ history comments
|
||
moves the needle.
|
||
2. **The best phrasing is model-dependent.** The terse one-liner `delete-stale`
|
||
is near-perfect on Opus (~94%) but only halfway on Haiku; the wordier
|
||
`drop-tombstones` is the reverse (75% Haiku, 67% Opus). Extra words help a
|
||
small model and distract a frontier one. We optimize for the model our agents
|
||
actually run on (Opus), so the one-liner wins — and it's the shorter line.
|
||
3. **The _add_ habit barely reproduces on modern models.** Earlier, weaker cases
|
||
(write fresh code; apply a clean rename) passed ~100% at baseline — the agents
|
||
almost never _add_ a change-narration comment unprompted. The habit only
|
||
surfaces under mimicry, when stale history comments already exist to copy.
|
||
|
||
`delete-stale` earned its line in the root `AGENTS.md`; the other phrasings did
|
||
not.
|
||
|
||
## Running
|
||
|
||
Requires the `claude` CLI on PATH, authenticated. Node 22+ runs the TypeScript
|
||
directly — no install step.
|
||
|
||
```bash
|
||
cd evals
|
||
pnpm eval # all cases, all candidates, 3 trials
|
||
TRIALS=5 node src/cli.ts # more trials = tighter signal
|
||
node src/cli.ts pr-review-comments # one case
|
||
CANDIDATES=baseline,describe-now node src/cli.ts # subset of candidates
|
||
AGENT_MODEL=claude-haiku-4-5 node src/cli.ts # pin the agent model
|
||
```
|
||
|
||
Results are printed and written to `results/latest.md`.
|
||
|
||
## Adding a case
|
||
|
||
Drop a file in `src/cases/` exporting an `EvalCase` and register it in
|
||
`src/cases/index.ts`. A good case has a `task` that tempts the mistake and a
|
||
`rubric` the judge can apply mechanically. Confirm `baseline` fails before
|
||
trusting any candidate that passes.
|
||
|
||
## Layout
|
||
|
||
```
|
||
src/
|
||
types.ts EvalCase / Candidate / Verdict
|
||
agent.ts runs the agent in a sandbox, with/without guidance
|
||
judge.ts scores an artifact against a rubric (LLM judge)
|
||
runner.ts baseline-vs-candidates A/B for one case
|
||
candidates.ts the guidance phrasings under test
|
||
cases/ the scenarios
|
||
cli.ts entry point
|
||
```
|