zzet--gortex/docs/04-evaluation/run.md

How to run

Operational recipe for executing the full methodology against a tagged build and publishing the results.

Prerequisites

• gortex binary built from the tagged commit
• python3 -m pip install -r eval/requirements.txt (Python harness deps; existing file)
• API keys: ANTHROPIC_API_KEY (for Sonnet 4.6), OPENAI_API_KEY (for GPT 5.4), and a working gh copilot installation (for Copilot CLI). At least one is enough for a partial run.
• A working corpus checkout (default: the gortex repo itself)

End-to-end

# 1) Tag the build — every published result cites this SHA.
git rev-parse HEAD > eval/results/$(date +%Y%m%d)/HEAD.sha

# 2) Run the full matrix: 15 tasks × 3 agents × 2 modes × 2 prompts.
#    Estimated wall clock: ~3-6 hours per agent.
gortex eval run \
    --task-set docs/04-evaluation/task-set.md \
    --judge-prompt docs/04-evaluation/judge-prompt.md \
    --agents sonnet-4.6,gpt-5.4,copilot \
    --corpus . \
    --out eval/results/$(date +%Y%m%d)/ \
    --max-task-tokens 50000 \
    --max-task-seconds 300

# 3) Aggregate per-task scores into the summary table.
python3 eval/scripts/aggregate.py \
    --workdir eval/results/$(date +%Y%m%d)/ \
    --judges sonnet-4.6,gpt-5.4 \
    --out eval/results/$(date +%Y%m%d)/summary.md

# 4) Spot-check 5 random tasks per category by hand BEFORE
#    publishing. The judge is good, not infallible.
python3 eval/scripts/spotcheck.py \
    --workdir eval/results/$(date +%Y%m%d)/ \
    --sample 5 \
    --out eval/results/$(date +%Y%m%d)/spotcheck.md

# 5) Promote into BENCHMARK.md (manual edit; doc owner).
$EDITOR BENCHMARK.md

What lands on disk

eval/results/<date>/
├── HEAD.sha                            # tagged commit
├── summary.md                          # the published table
├── spotcheck.md                        # manual review notes
├── disagreement.md                     # judge-vs-judge disagreement
├── per-task/
│   ├── 1.1-indexer-walkthrough/
│   │   ├── sonnet-4.6-with-default.json
│   │   ├── sonnet-4.6-without-default.json
│   │   ├── sonnet-4.6-with-ablation.json
│   │   ├── sonnet-4.6-without-ablation.json
│   │   ├── gpt-5.4-with-default.json
│   │   └── ...
│   ├── 1.2-community-detection/
│   └── ...
└── judge-runs/
    ├── sonnet-4.6-judging/
    └── gpt-5.4-judging/

Every per-task JSON contains: task prompt, canonical answer, agent answer, token cost, wall clock, tools called (count + list), and (if judged) the judge's label + reasoning + agreement between judges.

What to publish

In BENCHMARK.md, add a section like:

## Agent-graded evaluation

**Last run: 2026-MM-DD** · agents: Sonnet 4.6 / GPT 5.4 /
Copilot CLI · judge: Sonnet 4.6 + GPT 5.4 (agreement 87%)

| Category | (a) gortex helped | (b) no difference | (c) gortex hurt |
|----------|------------------:|------------------:|----------------:|
| Architectural explanation |  6 | 1 | 2 |
| Refactor safety           |  7 | 2 | 0 |
| Bug localization          |  5 | 2 | 2 |
| Impact analysis           |  8 | 1 | 0 |
| Contract extraction       |  6 | 3 | 0 |
| **Total**                 | 32 | 9 | 4 |

- Default-prompt vs ablation-prompt delta:  +2 (a) / 0 (b) / -1 (c)
  — gortex prompt steering helps but isn't load-bearing.
- (c) cases written up in `eval/results/2026-MM-DD/c-cases.md`
  — every loss has a public post-mortem.

Required: cite the (c) count, link to the (c) post-mortems, and call out the prompt-bias delta. A publication that hides (c) results is non-compliant with this methodology and should not be referenced as a benchmark.

Cost envelope

A single full run (15 tasks × 3 agents × 2 modes × 2 prompts = 180 agent runs + ~360 judge invocations) costs roughly:

• Anthropic API: ~$15-30 (Sonnet 4.6 agent + Sonnet 4.6 judge)
• OpenAI API: ~$10-25 (GPT 5.4 agent + GPT 5.4 judge)
• Copilot CLI: subscription-included

Total: ~$25-55 per full run. Run quarterly + on every major version bump.

Partial-run modes

When you only have one API key:

# Just Sonnet 4.6 (cheapest path)
gortex eval run --agents sonnet-4.6 --task-set ...

# Just one task category (smoke before the full run)
gortex eval run --agents sonnet-4.6 \
    --task-set docs/04-evaluation/task-set.md \
    --categories "Refactor safety"

# Just the WITH mode (compare absolute quality across agents)
gortex eval run --agents sonnet-4.6,gpt-5.4,copilot \
    --modes with

Partial runs are useful for iteration but don't publish partial-run numbers as benchmarks — the methodology requires the full matrix.