wehub-resource-sync
a06f331eb8
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
53 lines
2.3 KiB
Markdown
53 lines
2.3 KiB
Markdown
# Gortex evaluation methodology
|
|
|
|
This directory documents the agent-graded self-eval methodology
|
|
gortex uses to measure its own real-world quality. The headline:
|
|
we evaluate gortex with the agents that actually use it (Claude
|
|
Sonnet 4.6, GPT 5.4, Copilot CLI), on three real codebases, across
|
|
five task categories, with a documented judge prompt and an
|
|
explicit "report negative deltas" requirement.
|
|
|
|
This is methodology only — no published numbers live here. Numbers
|
|
land in [`BENCHMARK.md`](../../BENCHMARK.md) once we run the
|
|
methodology against a tagged build.
|
|
|
|
## Contents
|
|
|
|
- [`methodology.md`](methodology.md) — the protocol: agents, tasks,
|
|
classifiers, bias checks, negative-delta requirement.
|
|
- [`judge-prompt.md`](judge-prompt.md) — the exact judge prompt
|
|
template (reproducibility: change the prompt → bump the rev).
|
|
- [`task-set.md`](task-set.md) — the 15 seed tasks (3 per category)
|
|
with canonical answers.
|
|
- [`run.md`](run.md) — operational recipe: how to invoke the
|
|
harness, where outputs land, how to publish results.
|
|
|
|
## Why this exists
|
|
|
|
A retrieval / code-intelligence engine can ship excellent
|
|
substrate (graph, MCP tools, USD savings) and still produce
|
|
agents that prefer Read/Grep when given the choice. The eval
|
|
methodology answers "with our tools available, does the model
|
|
actually use them, and does it produce better answers than it
|
|
would without?" That's the real test — not benchmark NDCG@10
|
|
on synthetic queries.
|
|
|
|
The methodology has three properties competitors typically lack:
|
|
|
|
1. **Multi-agent**: same task set scored against ≥3 distinct
|
|
agent / model combinations so a result isn't a quirk of one
|
|
provider.
|
|
2. **Bias-of-prompt check**: every task runs with both the
|
|
default agent prompt AND a deliberately worse prompt (the
|
|
"ablation prompt"); a methodology that only looks good on the
|
|
tuned prompt is honest-flagged.
|
|
3. **Negative-delta requirement**: per-task scoring uses an
|
|
(a)/(b)/(c) classifier that distinguishes "gortex helped",
|
|
"no measurable difference", "gortex hurt". The published
|
|
summary MUST cite both ends — hiding the negatives gets the
|
|
methodology disqualified.
|
|
|
|
The substrate is already shipped (`gortex eval` substrate +
|
|
`eval/` Python harness); this directory makes it reproducible
|
|
end-to-end without an oral tradition.
|