wehub-resource-sync
113 lines
5.2 KiB
Markdown
113 lines
5.2 KiB
Markdown
# Golden-set eval for cheap-model routing (#1354)
|
||
|
||
> **Status: v1.** The *mechanism* (`eval-golden.mjs`) is design-invariant and runs
|
||
> today. Reference labels are now frozen (10 synthetic cases — see **Labeling
|
||
> methodology** below); the gate threshold and per-model cost remain tunable
|
||
> constants, and wiring into CI is still deferred (see **Open design questions**).
|
||
|
||
## What this is
|
||
|
||
A small labeled golden-set plus a harness that measures how well a *candidate*
|
||
cheap model agrees with reference labels, so "is model X good enough to route to?"
|
||
becomes a number instead of a hunch. It reuses the `---SCORE_SUMMARY---` contract
|
||
that every `*-eval.mjs` already emits (`SCORE` + `ARCHETYPE`), so there is no new
|
||
scoring surface.
|
||
|
||
## Labeling methodology (v1)
|
||
|
||
The metric is **agreement-with-reference, not absolute correctness.** For #1354's
|
||
purpose — *which cheap model can hold which task* — distance-to-reference is exactly
|
||
the right measure: if a candidate model reproduces the reference verdict, it is safe
|
||
to route that task to it.
|
||
|
||
**v1 labels are the frozen verdict of a reference (Claude-tier / premium) evaluation**
|
||
applied to each synthetic JD under the repo's own rubric (`modes/_shared.md` §
|
||
Scoring System + § Archetype Detection):
|
||
|
||
- **`archetype`** is the gate. It is a property of the JD alone (the 6-archetype
|
||
keyword taxonomy in `_shared.md`), so it is profile-independent and reproducible.
|
||
- **`score`** (1–5) is the reference model's fit verdict. It is profile-relative by
|
||
construction, but because both reference and candidate are scored under the *same*
|
||
conditions, distance-to-reference stays meaningful regardless of whose profile
|
||
drives it. It is a secondary, tolerance-banded signal — it does **not** gate.
|
||
|
||
The 10 cases cover all 6 archetypes and deliberately favor **edge archetypes over
|
||
easy wins**: four are hybrids/ambiguous (`platform-agentic-hybrid`,
|
||
`pm-architect-ambiguous`, `forward-deployed-vs-architect`, `transformation-vs-pm`)
|
||
where a cheaper model is most likely to diverge, and scores span 3.2–4.3 so the
|
||
tolerance band has real range to catch drift on red-flag postings.
|
||
|
||
**Future upgrade path:** hand-curated labels. Freezing the reference verdict is the
|
||
cheap, reproducible v1; a later pass can replace or reconcile individual labels with
|
||
human-graded ground truth (flip `provenance` to `hand-curated`) without changing the
|
||
harness.
|
||
|
||
## Layout
|
||
|
||
```
|
||
evals/
|
||
golden/ labeled cases — one JSON per case (synthetic JDs, no user data)
|
||
fixtures/ recorded candidate outputs for $0 deterministic replay in CI
|
||
README.md this file
|
||
eval-golden.mjs the harness (root level, sibling to openai-eval.mjs)
|
||
```
|
||
|
||
### Golden case format (`evals/golden/*.json`)
|
||
|
||
```json
|
||
{
|
||
"id": "ai-platform-llmops",
|
||
"synthetic": true,
|
||
"jd": "<full synthetic job description text>",
|
||
"label": { "archetype": "AI Platform / LLMOps", "score": 4.2, "provenance": "reference-frozen-v1" }
|
||
}
|
||
```
|
||
|
||
`provenance` records how the label was set (`reference-frozen-v1` today; future
|
||
hand-curated labels flip it). Edge cases may also carry an `edge_note` explaining
|
||
why the reference resolved an ambiguous JD the way it did. Both are advisory — the
|
||
harness only requires `archetype` (string) and `score` (number). All JDs are
|
||
**synthetic** so the set stays clear of the `no-user-data` guard.
|
||
|
||
### Fixture format (`evals/fixtures/<case-id>__<model>.txt`)
|
||
|
||
A recorded candidate-model output containing a `---SCORE_SUMMARY---` block. Only
|
||
that block is parsed; surrounding prose is illustrative and trimmed. Slash-form
|
||
provider ids (`deepseek/deepseek-chat`) are flattened to a path-safe token for the
|
||
filename (`<case>__deepseek-deepseek-chat.txt`), so a fixture never lands in a
|
||
phantom subdirectory.
|
||
|
||
## Running
|
||
|
||
```bash
|
||
npm run eval:golden -- --replay --model cheap-stub # offline, deterministic, $0
|
||
npm run eval:golden -- --live --model gpt-4o-mini # real call via openai-eval.mjs (needs key + cv.md)
|
||
```
|
||
|
||
Replay is the CI-friendly path: no API keys, no `cv.md`, fully deterministic.
|
||
The harness reports per-case archetype/score agreement, mean |Δscore|, median
|
||
latency (live only), and a placeholder $/run, then exits `0/1` on the archetype
|
||
agreement gate.
|
||
|
||
## Open design questions (TODO #1354)
|
||
|
||
Resolved in v1:
|
||
|
||
- **Reference labels** — frozen reference (Claude-tier) verdict; see **Labeling
|
||
methodology** above. Hand-curation is the documented future upgrade path.
|
||
- **Set size / spread** — grown from 2 to 10 cases across all 6 archetypes, favoring
|
||
edge/hybrid archetypes, scores spanning 3.2–4.3.
|
||
|
||
Still tunable (named constants, safe defaults today):
|
||
|
||
| Question | Where it lives | v1 default |
|
||
|----------|----------------|------------|
|
||
| `SCORE` agreement: tolerance band width | `SCORE_TOLERANCE` in `eval-golden.mjs` | ±0.5 (band, per distance-to-reference) |
|
||
| CI gate threshold for archetype agreement | `MIN_ARCHETYPE_AGREEMENT` in `eval-golden.mjs` | 0.8 |
|
||
| Per-model $/run rates | `COST_PER_RUN_USD` in `eval-golden.mjs` | empty — needs real provider rates |
|
||
|
||
Wiring this into the required CI job (`.github/workflows/test.yml`) is intentionally
|
||
deferred until the gate threshold is confirmed, so a default value can't make `main`
|
||
go red. The replay path is deterministic and $0, so it is ready to wire whenever the
|
||
threshold is signed off.
|