wehub-resource-sync
102 lines
4.5 KiB
Markdown
102 lines
4.5 KiB
Markdown
# 2026-05-20 — coding-agent-life-v1 (v0.9.26)
|
|
|
|
**Bench:** coding-agent-life-v1 (15 sessions, 15 queries)
|
|
**N:** 15
|
|
**K:** 5
|
|
**Hardware:** macOS 15 (Apple Silicon)
|
|
**agentmemory:** v0.9.26
|
|
**iii-engine:** v0.11.2
|
|
**Embedding provider:** local default
|
|
**Sandbox:** isolated data dir at `/tmp/agentmemory-eval-sandbox/`, ports 3411/3412
|
|
|
|
## Math ceiling on this dataset
|
|
|
|
12 of 15 questions have 1 gold session, 3 have 2 gold sessions. Per
|
|
`scoreQuestion` in `eval/runner/score.ts`, P@K = `hits / k` averaged
|
|
across questions, so the **maximum achievable P@5** is:
|
|
|
|
```
|
|
(12 * 1/5) + (3 * 2/5)) / 15 = (2.4 + 1.2) / 15 = 0.240
|
|
```
|
|
|
|
R@5 ceiling is **1.000** (every gold session found in top-5).
|
|
|
|
The benchmark is **small** (15 questions) and **gold-sparse** (mostly
|
|
single-gold). It's tuned for fast iteration on the retrieval stack,
|
|
not for headline P@K comparisons. **Recall** + per-question-type
|
|
**P@5** are the signals; aggregate P@5 saturates at 0.240 so it can't
|
|
differentiate top-tier adapters.
|
|
|
|
## Headline
|
|
|
|
`agentmemory-hybrid` hits **100% top-5 hit rate**, R@5 = **1.000**,
|
|
P@5 = **0.240** (at the math ceiling — every gold session retrieved
|
|
in top-5).
|
|
|
|
grep baseline: R@5 = **0.967**, P@5 = **0.227** — missed one gold
|
|
session in one multi-gold question. Lift is **recall**, not aggregate
|
|
precision.
|
|
|
|
## Per-adapter
|
|
|
|
| Adapter | P@5 | R@5 | Hit rate | p50 latency |
|
|
|---|---|---|---|---|
|
|
| grep (tokenized substring) | 0.227 | 0.967 | 15 / 15 | 0 ms |
|
|
| `agentmemory-hybrid` | **0.240** | **1.000** | **15 / 15** | 14 ms |
|
|
|
|
`agentmemory-hybrid` runs through the production smart-search endpoint (`POST /agentmemory/smart-search`) so it exercises the full BM25 + embedding + reranker stack.
|
|
|
|
## Per-question-type
|
|
|
|
At K=5 with 1 gold per single-session question, the P@5 ceiling per
|
|
question is **0.20**; with 2 gold the ceiling is **0.40**. Both
|
|
adapters saturate the per-type ceiling on most types, so the per-type
|
|
table primarily exposes where one adapter **misses** gold (failing
|
|
the recall side).
|
|
|
|
| Type | grep P@5 | grep R@5 | hybrid P@5 | hybrid R@5 |
|
|
|---|---|---|---|---|
|
|
| single-session-bug | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-infra (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-refactor | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-feature | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-test | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-perf | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-api | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-db | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| single-session-release | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| multi-session-causal (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
|
|
| preference (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
|
|
| multi-session-review (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
|
|
| temporal (2 gold) | 0.20 | 0.50 | 0.40 | 1.00 |
|
|
|
|
The differentiator at this corpus size is **temporal** (`What was shipped on April 8th 2026?`): grep finds 1 of 2 gold sessions, hybrid finds both. Per-type R@5 saturates everywhere else.
|
|
|
|
## Methodology
|
|
|
|
- 15 fictional Claude Code sessions across a 10-day stretch of a Rust CLI project (`shipctl`) — bug fixes, refactors, infra, perf, schema migrations, preferences, post-mortem
|
|
- 15 hand-graded queries with `goldSessionIds[]` covering single-session, multi-session causal, multi-session review, preference, temporal
|
|
- Each session ingested via `POST /agentmemory/remember` with `type=eval-session` and `concepts=[session_id]`
|
|
- Each query hits `POST /agentmemory/smart-search` with `limit=50`; dedupe by session ID; truncate to K=5
|
|
- No LLM in the retrieval loop
|
|
- Sandbox: clean `~/.agentmemory` via `HOME` override + alt ports (3411/3412) so no cross-contamination from a user's real store
|
|
|
|
## Reproduce
|
|
|
|
```sh
|
|
git checkout e9dc710
|
|
npm install --legacy-peer-deps
|
|
npm run build
|
|
|
|
source eval/scripts/sandbox.sh
|
|
npm run eval:coding-life -- --adapters grep,agentmemory
|
|
```
|
|
|
|
Outputs land in `eval/reports/coding-life/`: `scores.ndjson` (per-query rows) and `summary.json` (per-adapter and per-type aggregates).
|
|
|
|
## Notes
|
|
|
|
- The single-session-feature tie (`Which PR introduced helm chart support?`) is interesting: query says `PR introduced helm chart` and gold session has `helm chart` literally — grep wins on lexical exactness, hybrid matches but doesn't outperform.
|
|
- The corpus is intentionally small for fast iteration. Hardening targets: paraphrased queries, synonym substitution, in-corpus distractors with shared keywords, longer multi-session chains.
|
|
- Vector adapter not measured here — requires `OPENAI_API_KEY`; will be added in a follow-up scorecard alongside LongMemEval `_s`.
|