wehub-resource-sync
26382a7ac6
CI / Clippy (push) Failing after 15m13s
CI / Test (ubuntu-latest) (push) Failing after 16m1s
CI / Test (macos-latest) (push) Has been cancelled
CI / Test (windows-latest) (push) Has been cancelled
CI / Build (no embeddings / no ORT) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Cookbook (Node) (push) Has been cancelled
CI / Pi Extension (Node) (push) Has been cancelled
CI / Rust SDK (lean-ctx-client) (push) Has been cancelled
CI / Embed SDK (lean-ctx-sdk) (push) Has been cancelled
CI / Python SDK (leanctx) (push) Has been cancelled
CI / Hermes Plugin (Python) (push) Has been cancelled
CI / SDK Conformance Matrix (push) Has been cancelled
CI / Coverage (push) Has been cancelled
CI / cargo-deny (push) Has been cancelled
CI / Adversarial Safety (push) Has been cancelled
CI / Benchmarks (push) Has been cancelled
CI / Output-Quality Gate (eval A/B) (push) Has been cancelled
CI / Documentation (push) Has been cancelled
CI / CI Green (push) Has been cancelled
JetBrains Plugin / Actionlint (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (rust) (push) Has been cancelled
JetBrains Plugin / Validation (push) Has been cancelled
JetBrains Plugin / Build (push) Has been cancelled
JetBrains Plugin / Test (push) Has been cancelled
Security Check / Security Scan (push) Has been cancelled
79 lines
3.6 KiB
Markdown
79 lines
3.6 KiB
Markdown
# Agent-Task-Benchmark v1 (GL #493)
|
||
|
||
Outcome evidence, not token arithmetic: **does lean-ctx change task success
|
||
rate and cost per solved task** for an agentic coding workload? Two identical
|
||
arms (Claude Code headless, native vs. lean-ctx MCP) over a deterministic
|
||
SWE-bench-Verified subset, judged by the official SWE-bench evaluation.
|
||
|
||
The methodology is pre-registered in [`PROTOCOL.md`](PROTOCOL.md) — read it
|
||
first; it is frozen, and changes require numbered amendments. Negative or
|
||
neutral results are published unchanged.
|
||
|
||
## Prerequisites
|
||
|
||
- Python ≥ 3.9, `pip install -r requirements.txt`
|
||
- `claude` (Claude Code CLI) on PATH, `ANTHROPIC_API_KEY` exported
|
||
- `lean-ctx` on PATH (pinned release; version is recorded per run)
|
||
- Docker (only for the evaluation step)
|
||
- Disk: repo mirrors (~2 GB) + SWE-bench evaluation images
|
||
|
||
> Safety: agent runs execute model-chosen shell commands with permissions
|
||
> skipped (standard for SWE-bench harnesses). Run on a disposable machine or
|
||
> container, not on a workstation with credentials you care about.
|
||
|
||
## Runbook
|
||
|
||
```bash
|
||
cd bench/agent-task
|
||
|
||
# 0a. Preflight — verify binaries, auth, network, disk BEFORE spending money
|
||
python3 -m swebench_harness.preflight # add --evaluate before step 3
|
||
|
||
# 0b. Task set: already frozen as tasks.lock.json (n=15, 12 repos) — do NOT
|
||
# regenerate for v1 (select_tasks refuses if the lock exists).
|
||
|
||
# 1. Smoke (1 instance, both arms) — validates plumbing end to end
|
||
python3 -m swebench_harness.run_arm --arm native --run-id smoke --instance <id-from-lock>
|
||
python3 -m swebench_harness.run_arm --arm leanctx --run-id smoke --instance <id-from-lock>
|
||
|
||
# 2. Full run (N=15 × 2 arms; resumable — finished runs are skipped)
|
||
python3 -m swebench_harness.run_arm --arm native --run-id v1
|
||
python3 -m swebench_harness.run_arm --arm leanctx --run-id v1
|
||
|
||
# 3. Predictions + official evaluation (docker)
|
||
python3 -m swebench_harness.collect --run-id v1
|
||
python3 -m swebench.harness.run_evaluation \
|
||
--dataset_name princeton-nlp/SWE-bench_Verified \
|
||
--predictions_path runs/v1/predictions-native.jsonl --run_id v1-native --max_workers 4
|
||
python3 -m swebench.harness.run_evaluation \
|
||
--dataset_name princeton-nlp/SWE-bench_Verified \
|
||
--predictions_path runs/v1/predictions-leanctx.jsonl --run_id v1-leanctx --max_workers 4
|
||
|
||
# 4. Endpoints + verifiable artifact (+ optional signature)
|
||
python3 -m swebench_harness.report --run-id v1 \
|
||
--eval-report native=claude-code-native.v1-native.json \
|
||
--eval-report leanctx=claude-code-leanctx.v1-leanctx.json
|
||
```
|
||
|
||
## What gets recorded
|
||
|
||
| Artifact | Contents |
|
||
|---|---|
|
||
| `runs/<id>/<instance>/<arm>/transcript.jsonl` | full agent stream (turns, tool calls, usage) |
|
||
| `…/model_patch.diff` | the submitted patch (`git add -A && git diff --cached`) |
|
||
| `…/meta.json` | exit code, wall time, billed tokens, cost, versions, flags |
|
||
| `runs/<id>/predictions-<arm>.jsonl` | official SWE-bench prediction format |
|
||
| `runs/<id>/result-v1.json` | canonical result artifact, self-hashing, embeds protocol + lock SHA-256 |
|
||
| `runs/<id>/result-v1.md` | human-readable endpoint table |
|
||
|
||
## Design notes
|
||
|
||
- **Fresh HOME per run** — the operator's machine has lean-ctx globally
|
||
installed; without isolation the native arm would be contaminated.
|
||
- **`--strict-mcp-config`** pins the MCP surface per arm explicitly: empty for
|
||
native, exactly the `lean-ctx init --agent claude` output for leanctx.
|
||
- **Usage comes from the runtime's own final report** (stream-json `result`
|
||
event) — no token estimation anywhere.
|
||
- **Selection is rule-based, not random** (sorted round-robin by repo), so the
|
||
subset is reproducible from the public dataset without a seed argument.
|