wehub-resource-sync
26382a7ac6
CI / Clippy (push) Failing after 15m13s
CI / Test (ubuntu-latest) (push) Failing after 16m1s
CI / Test (macos-latest) (push) Has been cancelled
CI / Test (windows-latest) (push) Has been cancelled
CI / Build (no embeddings / no ORT) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Cookbook (Node) (push) Has been cancelled
CI / Pi Extension (Node) (push) Has been cancelled
CI / Rust SDK (lean-ctx-client) (push) Has been cancelled
CI / Embed SDK (lean-ctx-sdk) (push) Has been cancelled
CI / Python SDK (leanctx) (push) Has been cancelled
CI / Hermes Plugin (Python) (push) Has been cancelled
CI / SDK Conformance Matrix (push) Has been cancelled
CI / Coverage (push) Has been cancelled
CI / cargo-deny (push) Has been cancelled
CI / Adversarial Safety (push) Has been cancelled
CI / Benchmarks (push) Has been cancelled
CI / Output-Quality Gate (eval A/B) (push) Has been cancelled
CI / Documentation (push) Has been cancelled
CI / CI Green (push) Has been cancelled
JetBrains Plugin / Actionlint (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (rust) (push) Has been cancelled
JetBrains Plugin / Validation (push) Has been cancelled
JetBrains Plugin / Build (push) Has been cancelled
JetBrains Plugin / Test (push) Has been cancelled
Security Check / Security Scan (push) Has been cancelled
94 lines
4.2 KiB
Markdown
94 lines
4.2 KiB
Markdown
# Agent-Task-Benchmark v1 — Pre-Registered Protocol (GL #493)
|
|
|
|
Status: **FROZEN at commit time.** Changes after the first recorded run require a
|
|
numbered amendment (`A1`, `A2`, …) appended under *Amendments* — the original
|
|
text is never edited. This mirrors the standard set by the external tokbench
|
|
study (GH #361): protocol first, runs second, publication third.
|
|
|
|
## 1. Question
|
|
|
|
Does lean-ctx change the **outcome** of an agentic coding workload — task
|
|
success rate and cost per solved task — compared to the identical agent
|
|
without lean-ctx?
|
|
|
|
This deliberately measures outcome, not token arithmetic. Token deltas are
|
|
reported, but the primary endpoints are:
|
|
|
|
- **resolved rate** (per the official SWE-bench evaluation harness), and
|
|
- **cost per resolved task** (billed USD / resolved count).
|
|
|
|
## 2. Workload
|
|
|
|
- **Dataset:** `princeton-nlp/SWE-bench_Verified` (test split), the
|
|
human-validated 500-instance subset.
|
|
- **Subset size:** N = 15 instances.
|
|
- **Selection rule (deterministic, no cherry-picking):** sort all instances by
|
|
`instance_id` ascending; group by `repo`; visit repos in ascending name
|
|
order round-robin, taking the lexicographically first untaken instance from
|
|
each repo per cycle, until N instances are selected. The result is committed
|
|
as `tasks.lock.json` (content-hashed into the result artifact). The lock is
|
|
generated once by `select_tasks.py` and never regenerated for v1.
|
|
- Instances are independent; each run starts from a clean checkout of
|
|
`base_commit`.
|
|
|
|
## 3. Arms
|
|
|
|
Two arms, identical in every respect except lean-ctx:
|
|
|
|
| | `native` | `leanctx` |
|
|
|---|---|---|
|
|
| Agent | Claude Code headless (`claude -p`), pinned version recorded in `meta.json` | identical |
|
|
| Prompt | `PROMPT.md` template, identical text | identical |
|
|
| MCP config | none (`--strict-mcp-config` with empty config) | exactly the registration `lean-ctx init --agent claude` writes, extracted into an explicit config and pinned via `--strict-mcp-config`; missing registration aborts the run; lean-ctx version recorded |
|
|
| Rules file | none | `CLAUDE.md`/rules as written by `lean-ctx init` (stock; no hand-tuning) |
|
|
| HOME | fresh per-run temp HOME (no user-level config bleed) | identical |
|
|
| Max turns | 40 | identical |
|
|
| Model | the agent runtime's pinned default model, recorded per run | identical |
|
|
|
|
The fresh-HOME isolation exists because the operator's real machine has
|
|
lean-ctx globally installed; without it the `native` arm would be
|
|
contaminated.
|
|
|
|
## 4. Measurement
|
|
|
|
- **Patch:** after the agent exits, `git add -A && git diff --cached` in the
|
|
task workspace is the submitted `model_patch`.
|
|
- **Resolution:** official `swebench.harness.run_evaluation` (dockerized) on
|
|
the per-arm `predictions.jsonl`. An instance counts as resolved iff the
|
|
official report lists it as resolved. No manual judging.
|
|
- **Tokens & cost:** taken from the agent runtime's own final usage report
|
|
per run (stream-json `result` event: input/output/cache tokens,
|
|
`total_cost_usd`). We do not estimate; if the runtime reports no usage the
|
|
run is marked `usage_missing` and excluded from cost endpoints (counted in
|
|
resolution endpoints).
|
|
- **Wall time:** harness-measured per run.
|
|
|
|
## 5. Endpoints
|
|
|
|
Primary:
|
|
1. resolved-rate per arm (resolved / N),
|
|
2. cost per resolved task per arm (sum billed USD / resolved).
|
|
|
|
Secondary: billed input tokens per run (median), output tokens, wall time,
|
|
turns used, lean-ctx tool-call adoption in the `leanctx` arm (from
|
|
transcripts).
|
|
|
|
## 6. Honesty constraints
|
|
|
|
- Both arms run from the same task lock, same prompt, same limits.
|
|
- All transcripts (`transcript.jsonl`), patches and the evaluation report are
|
|
retained as raw artifacts and published alongside the result.
|
|
- Negative or neutral results are published unchanged.
|
|
- The result artifact embeds the SHA-256 of this protocol and of
|
|
`tasks.lock.json`; `report.py` emits the artifact digest for signing
|
|
(`ssh-keygen -Y sign`) so third parties can verify nothing moved after the
|
|
fact.
|
|
- Known limitation, stated up front: N=15 with one seed is a pilot-grade
|
|
sample — confidence intervals are wide; the claim is directional, not a
|
|
leaderboard. Provider-side prompt caching is active for both arms equally
|
|
(it is part of the product reality being measured).
|
|
|
|
## 7. Amendments
|
|
|
|
*(none yet)*
|