zzet--gortex/BENCHMARK-SWE.md

Gortex on SWE-bench

This document is the public results template for SWE-bench runs against gortex's MCP-driven agent. The harness lives at cmd/gortex/eval_swebench.go and eval/ (the Python side); running it end-to-end takes multi-day GPU compute on the full benchmark, so we ship the template + reproducibility instructions here and update the numbers section after each substantive run.

Results

Last run: TBD — see the "How to reproduce" section to run it yourself; replace this section with your numbers afterward.

model	benchmark variant	n_resolved	n_total	resolve_rate	avg tokens	avg runtime
TBD	SWE-bench Lite	—	—	—	—	—
TBD	SWE-bench Verified	—	—	—	—	—
TBD	SWE-bench	—	—	—	—	—

When a row populates: include the exact model name (e.g. claude-sonnet-4-20250514), the harness commit SHA, the run date, and the model card the per-task prompts use. Append a results/swebench/<run-id>/ directory with per-task JSON + the overall summary so any reviewer can spot-check the count.

Methodology

Gortex's SWE-bench harness is a thin agent that exposes the same MCP tool surface as a regular session (smart_context, search_symbols, get_symbol_source, edit_file, verify_change, …) and lets the configured LLM provider drive a turn loop. Per-task budget is the same token / wall-clock cap as the upstream SWE-bench harness so results are comparable to other published numbers.

The runner persists per-task outputs to results/swebench/<run-id>/<task-id>/ so a failed task can be re-played without re-running the whole benchmark.

Honest caveats:

• Compute envelope. The full SWE-bench (~2300 tasks) takes multi-day GPU compute even at modest concurrency; SWE-bench Lite (300 tasks) is the practical target for iteration. Don't publish "full SWE-bench" numbers without showing the run-time cost too.
• Dataset license. SWE-bench is community-maintained; check the upstream licence before redistributing the per-task artifacts.
• Per-model variance. Run-to-run variance is non-trivial (~2-5 percentage points on resolve rate); a published number is one sample, not a confidence interval. Re-run before citing.

How to reproduce

# 1) Pre-flight: ensure the harness substrate is in place.
ls eval/                       # Python harness lives here
ls cmd/gortex/eval_swebench.go # Go-side CLI entry

# 2) List available SWE-bench configurations (Lite / Verified /
#    full / custom subsets).
gortex eval swebench --list-configs

# 3) Run a small smoke against SWE-bench Lite, default config.
gortex eval swebench \
    --config swebench-lite \
    --model claude-sonnet-4-20250514 \
    --workdir results/swebench/$(date +%Y%m%d-%H%M%S)/ \
    --max-tasks 5

# 4) Full-config run (multi-day; only do this when you mean it).
gortex eval swebench \
    --config swebench-lite \
    --model claude-sonnet-4-20250514 \
    --workdir results/swebench/$(date +%Y%m%d-%H%M%S)/

# 5) Aggregate the per-task JSON into a summary row.
python3 eval/scripts/aggregate_swebench.py \
    --workdir results/swebench/<run-id>/ \
    --out results/swebench/<run-id>/summary.json

# 6) Paste the numbers into the table above; commit results/.

See eval/README.md for the Python-side configuration options (per-task token budgets, retry policy, judge model, etc.).

Cross-links

• Other reproducible benchmarks: BENCHMARK.md
• Evaluation methodology: docs/04-evaluation/ (when shipped)
• Substrate: cmd/gortex/eval_swebench.go + eval/