zzet--gortex/docs/04-evaluation/methodology.md

Methodology

The protocol below produces reproducible, agent-graded scoring of gortex's real-world effect on coding tasks. Sticking to the protocol means anyone with the harness + a model API key can reproduce the numbers and dispute them.

1. Task categories (5)

Category	What it measures	Example
Architectural explanation	"Why does this codebase have N services" — graph / community structure understanding	"Walk me through how the indexer pipeline processes a new file."
Refactor safety	"Rename / move / extract — what breaks?" — impact-analysis driven	"Rename Indexer.Index to Indexer.IndexRoot across the repo. List every caller that needs updating."
Bug localization	"Given this failure, where's the root cause?" — call-chain + dataflow	"A panic in internal/savings/store.go::flushLocked. Trace the conditions that reach it."
Impact analysis	"If I change X, what tests should I run?" — get_test_targets + verify_change	"I'm about to change the signature of tokens.Count. List the test files that need re-running."
Contract extraction	"What's the public API surface of this package?" — contracts list driven	"List every exported function / method / type in internal/savings/, with one-line summaries."

Each task is realistic — drawn from actual sessions, not invented synthetic prompts. The seed task set in task-set.md ships 3 tasks per category × 5 categories = 15 tasks.

2. Agent / model matrix

The same task set runs against each of:

1. Claude Sonnet 4.6 via the Anthropic API (claude-sonnet-4-20250514)
2. GPT 5.4 via OpenAI (gpt-5-2025-08)
3. Copilot CLI via the GitHub CLI extension

For each agent × task combination, two runs:

• WITH gortex MCP — the agent has access to the full gortex tool surface (smart_context, search_symbols, get_symbol_source, verify_change, …)
• WITHOUT gortex MCP — the agent has only its default tool set (typically Read, Grep, Bash)

So per task: 6 runs (3 agents × 2 modes). Per task set: 90 runs. Per category: 18 runs.

3. Bias-of-prompt check

Each WITH-gortex run is executed twice:

• default prompt — the system prompt gortex ships in internal/agents/instructions.go (the same one the production gortex init writes to every agent's config)
• ablation prompt — the same prompt with every "prefer gortex tools" steering line removed

If the published headline only shows the default-prompt number, the methodology is incomplete. Always publish both, and call out the delta — that's the "we're not just measuring the prompt" test.

4. (a) / (b) / (c) classifier

A judge model (default Claude Sonnet 4.6 — see judge-prompt.md) scores each per-task WITH-vs-WITHOUT comparison with one of three labels:

• (a) gortex helped — the WITH run produced a measurably better answer (more accurate, more complete, fewer hallucinations, or substantially fewer tokens)
• (b) no measurable difference — answers are roughly equivalent in quality and cost
• (c) gortex hurt — the WITH run was worse (less accurate, more confused by the tool surface, or noticeably more tokens for the same answer)

The published summary MUST report all three counts. A "(a)=12 / (b)=2 / (c)=1" result is honest; "12 wins" is not.

5. Negative-delta requirement

Negative deltas (any (c) result) are required in the published summary, with the per-task breakdown linkable. The explicit requirement is the methodology's anti-survivorship mechanism: a methodology that buries (c) cases isn't measuring real-world quality, it's measuring marketing.

If a published run reports zero (c) results across all 15 tasks, that's a red flag — either the judge is biased or the seed set is over-fit. Re-run with a different judge model and at least 3 additional tasks per category before publishing.

6. Scoring envelope

Per-task token + wall-clock budgets are the same WITH and WITHOUT (typically 50k tokens / 5 minutes per task). A run that exceeds the budget scores as "no answer" — not as (c). This keeps the comparison about quality of answers, not endurance.

Per-task cost is reported separately so a published row can show "answer (a) at 1.2× the WITHOUT cost"; (a) at 5× the cost is honestly weaker than (a) at 0.8× the cost.

7. What we don't measure here

• Benchmark NDCG / recall — covered by bench/baselines/ and bench/token-efficiency/. Different axis: those measure retrieval quality independently of agent / model; this methodology measures real agent behaviour.
• SWE-bench resolve rate — covered by BENCHMARK-SWE.md. Multi-day GPU compute; published separately when an operator runs it.
• Performance — covered by bench/perf/. Indexer / query / impact latency, not answer quality.

Together the three axes (retrieval / agent behaviour / system perf) form gortex's published-quality envelope.