chore: import upstream snapshot with attribution
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
# How to run
|
||||
|
||||
Operational recipe for executing the full methodology against a
|
||||
tagged build and publishing the results.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `gortex` binary built from the tagged commit
|
||||
- `python3 -m pip install -r eval/requirements.txt` (Python
|
||||
harness deps; existing file)
|
||||
- API keys: `ANTHROPIC_API_KEY` (for Sonnet 4.6),
|
||||
`OPENAI_API_KEY` (for GPT 5.4), and a working `gh copilot`
|
||||
installation (for Copilot CLI). At least one is enough for a
|
||||
partial run.
|
||||
- A working corpus checkout (default: the gortex repo itself)
|
||||
|
||||
## End-to-end
|
||||
|
||||
```sh
|
||||
# 1) Tag the build — every published result cites this SHA.
|
||||
git rev-parse HEAD > eval/results/$(date +%Y%m%d)/HEAD.sha
|
||||
|
||||
# 2) Run the full matrix: 15 tasks × 3 agents × 2 modes × 2 prompts.
|
||||
# Estimated wall clock: ~3-6 hours per agent.
|
||||
gortex eval run \
|
||||
--task-set docs/04-evaluation/task-set.md \
|
||||
--judge-prompt docs/04-evaluation/judge-prompt.md \
|
||||
--agents sonnet-4.6,gpt-5.4,copilot \
|
||||
--corpus . \
|
||||
--out eval/results/$(date +%Y%m%d)/ \
|
||||
--max-task-tokens 50000 \
|
||||
--max-task-seconds 300
|
||||
|
||||
# 3) Aggregate per-task scores into the summary table.
|
||||
python3 eval/scripts/aggregate.py \
|
||||
--workdir eval/results/$(date +%Y%m%d)/ \
|
||||
--judges sonnet-4.6,gpt-5.4 \
|
||||
--out eval/results/$(date +%Y%m%d)/summary.md
|
||||
|
||||
# 4) Spot-check 5 random tasks per category by hand BEFORE
|
||||
# publishing. The judge is good, not infallible.
|
||||
python3 eval/scripts/spotcheck.py \
|
||||
--workdir eval/results/$(date +%Y%m%d)/ \
|
||||
--sample 5 \
|
||||
--out eval/results/$(date +%Y%m%d)/spotcheck.md
|
||||
|
||||
# 5) Promote into BENCHMARK.md (manual edit; doc owner).
|
||||
$EDITOR BENCHMARK.md
|
||||
```
|
||||
|
||||
## What lands on disk
|
||||
|
||||
```
|
||||
eval/results/<date>/
|
||||
├── HEAD.sha # tagged commit
|
||||
├── summary.md # the published table
|
||||
├── spotcheck.md # manual review notes
|
||||
├── disagreement.md # judge-vs-judge disagreement
|
||||
├── per-task/
|
||||
│ ├── 1.1-indexer-walkthrough/
|
||||
│ │ ├── sonnet-4.6-with-default.json
|
||||
│ │ ├── sonnet-4.6-without-default.json
|
||||
│ │ ├── sonnet-4.6-with-ablation.json
|
||||
│ │ ├── sonnet-4.6-without-ablation.json
|
||||
│ │ ├── gpt-5.4-with-default.json
|
||||
│ │ └── ...
|
||||
│ ├── 1.2-community-detection/
|
||||
│ └── ...
|
||||
└── judge-runs/
|
||||
├── sonnet-4.6-judging/
|
||||
└── gpt-5.4-judging/
|
||||
```
|
||||
|
||||
Every per-task JSON contains: task prompt, canonical answer,
|
||||
agent answer, token cost, wall clock, tools called (count +
|
||||
list), and (if judged) the judge's label + reasoning + agreement
|
||||
between judges.
|
||||
|
||||
## What to publish
|
||||
|
||||
In `BENCHMARK.md`, add a section like:
|
||||
|
||||
```markdown
|
||||
## Agent-graded evaluation
|
||||
|
||||
**Last run: 2026-MM-DD** · agents: Sonnet 4.6 / GPT 5.4 /
|
||||
Copilot CLI · judge: Sonnet 4.6 + GPT 5.4 (agreement 87%)
|
||||
|
||||
| Category | (a) gortex helped | (b) no difference | (c) gortex hurt |
|
||||
|----------|------------------:|------------------:|----------------:|
|
||||
| Architectural explanation | 6 | 1 | 2 |
|
||||
| Refactor safety | 7 | 2 | 0 |
|
||||
| Bug localization | 5 | 2 | 2 |
|
||||
| Impact analysis | 8 | 1 | 0 |
|
||||
| Contract extraction | 6 | 3 | 0 |
|
||||
| **Total** | 32 | 9 | 4 |
|
||||
|
||||
- Default-prompt vs ablation-prompt delta: +2 (a) / 0 (b) / -1 (c)
|
||||
— gortex prompt steering helps but isn't load-bearing.
|
||||
- (c) cases written up in `eval/results/2026-MM-DD/c-cases.md`
|
||||
— every loss has a public post-mortem.
|
||||
```
|
||||
|
||||
**Required**: cite the (c) count, link to the (c)
|
||||
post-mortems, and call out the prompt-bias delta. A publication
|
||||
that hides (c) results is non-compliant with this methodology
|
||||
and should not be referenced as a benchmark.
|
||||
|
||||
## Cost envelope
|
||||
|
||||
A single full run (15 tasks × 3 agents × 2 modes × 2 prompts =
|
||||
180 agent runs + ~360 judge invocations) costs roughly:
|
||||
|
||||
- Anthropic API: ~$15-30 (Sonnet 4.6 agent + Sonnet 4.6 judge)
|
||||
- OpenAI API: ~$10-25 (GPT 5.4 agent + GPT 5.4 judge)
|
||||
- Copilot CLI: subscription-included
|
||||
|
||||
Total: ~$25-55 per full run. Run quarterly + on every major
|
||||
version bump.
|
||||
|
||||
## Partial-run modes
|
||||
|
||||
When you only have one API key:
|
||||
|
||||
```sh
|
||||
# Just Sonnet 4.6 (cheapest path)
|
||||
gortex eval run --agents sonnet-4.6 --task-set ...
|
||||
|
||||
# Just one task category (smoke before the full run)
|
||||
gortex eval run --agents sonnet-4.6 \
|
||||
--task-set docs/04-evaluation/task-set.md \
|
||||
--categories "Refactor safety"
|
||||
|
||||
# Just the WITH mode (compare absolute quality across agents)
|
||||
gortex eval run --agents sonnet-4.6,gpt-5.4,copilot \
|
||||
--modes with
|
||||
```
|
||||
|
||||
Partial runs are useful for iteration but **don't publish
|
||||
partial-run numbers as benchmarks** — the methodology requires
|
||||
the full matrix.
|
||||
Reference in New Issue
Block a user