chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,101 @@
|
||||
# 2026-05-20 — coding-agent-life-v1 (v0.9.26)
|
||||
|
||||
**Bench:** coding-agent-life-v1 (15 sessions, 15 queries)
|
||||
**N:** 15
|
||||
**K:** 5
|
||||
**Hardware:** macOS 15 (Apple Silicon)
|
||||
**agentmemory:** v0.9.26
|
||||
**iii-engine:** v0.11.2
|
||||
**Embedding provider:** local default
|
||||
**Sandbox:** isolated data dir at `/tmp/agentmemory-eval-sandbox/`, ports 3411/3412
|
||||
|
||||
## Math ceiling on this dataset
|
||||
|
||||
12 of 15 questions have 1 gold session, 3 have 2 gold sessions. Per
|
||||
`scoreQuestion` in `eval/runner/score.ts`, P@K = `hits / k` averaged
|
||||
across questions, so the **maximum achievable P@5** is:
|
||||
|
||||
```
|
||||
(12 * 1/5) + (3 * 2/5)) / 15 = (2.4 + 1.2) / 15 = 0.240
|
||||
```
|
||||
|
||||
R@5 ceiling is **1.000** (every gold session found in top-5).
|
||||
|
||||
The benchmark is **small** (15 questions) and **gold-sparse** (mostly
|
||||
single-gold). It's tuned for fast iteration on the retrieval stack,
|
||||
not for headline P@K comparisons. **Recall** + per-question-type
|
||||
**P@5** are the signals; aggregate P@5 saturates at 0.240 so it can't
|
||||
differentiate top-tier adapters.
|
||||
|
||||
## Headline
|
||||
|
||||
`agentmemory-hybrid` hits **100% top-5 hit rate**, R@5 = **1.000**,
|
||||
P@5 = **0.240** (at the math ceiling — every gold session retrieved
|
||||
in top-5).
|
||||
|
||||
grep baseline: R@5 = **0.967**, P@5 = **0.227** — missed one gold
|
||||
session in one multi-gold question. Lift is **recall**, not aggregate
|
||||
precision.
|
||||
|
||||
## Per-adapter
|
||||
|
||||
| Adapter | P@5 | R@5 | Hit rate | p50 latency |
|
||||
|---|---|---|---|---|
|
||||
| grep (tokenized substring) | 0.227 | 0.967 | 15 / 15 | 0 ms |
|
||||
| `agentmemory-hybrid` | **0.240** | **1.000** | **15 / 15** | 14 ms |
|
||||
|
||||
`agentmemory-hybrid` runs through the production smart-search endpoint (`POST /agentmemory/smart-search`) so it exercises the full BM25 + embedding + reranker stack.
|
||||
|
||||
## Per-question-type
|
||||
|
||||
At K=5 with 1 gold per single-session question, the P@5 ceiling per
|
||||
question is **0.20**; with 2 gold the ceiling is **0.40**. Both
|
||||
adapters saturate the per-type ceiling on most types, so the per-type
|
||||
table primarily exposes where one adapter **misses** gold (failing
|
||||
the recall side).
|
||||
|
||||
| Type | grep P@5 | grep R@5 | hybrid P@5 | hybrid R@5 |
|
||||
|---|---|---|---|---|
|
||||
| single-session-bug | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-infra (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-refactor | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-feature | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-test | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-perf | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-api | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-db | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| single-session-release | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| multi-session-causal (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
|
||||
| preference (n=2) | 0.20 | 1.00 | 0.20 | 1.00 |
|
||||
| multi-session-review (2 gold) | 0.40 | 1.00 | 0.40 | 1.00 |
|
||||
| temporal (2 gold) | 0.20 | 0.50 | 0.40 | 1.00 |
|
||||
|
||||
The differentiator at this corpus size is **temporal** (`What was shipped on April 8th 2026?`): grep finds 1 of 2 gold sessions, hybrid finds both. Per-type R@5 saturates everywhere else.
|
||||
|
||||
## Methodology
|
||||
|
||||
- 15 fictional Claude Code sessions across a 10-day stretch of a Rust CLI project (`shipctl`) — bug fixes, refactors, infra, perf, schema migrations, preferences, post-mortem
|
||||
- 15 hand-graded queries with `goldSessionIds[]` covering single-session, multi-session causal, multi-session review, preference, temporal
|
||||
- Each session ingested via `POST /agentmemory/remember` with `type=eval-session` and `concepts=[session_id]`
|
||||
- Each query hits `POST /agentmemory/smart-search` with `limit=50`; dedupe by session ID; truncate to K=5
|
||||
- No LLM in the retrieval loop
|
||||
- Sandbox: clean `~/.agentmemory` via `HOME` override + alt ports (3411/3412) so no cross-contamination from a user's real store
|
||||
|
||||
## Reproduce
|
||||
|
||||
```sh
|
||||
git checkout e9dc710
|
||||
npm install --legacy-peer-deps
|
||||
npm run build
|
||||
|
||||
source eval/scripts/sandbox.sh
|
||||
npm run eval:coding-life -- --adapters grep,agentmemory
|
||||
```
|
||||
|
||||
Outputs land in `eval/reports/coding-life/`: `scores.ndjson` (per-query rows) and `summary.json` (per-adapter and per-type aggregates).
|
||||
|
||||
## Notes
|
||||
|
||||
- The single-session-feature tie (`Which PR introduced helm chart support?`) is interesting: query says `PR introduced helm chart` and gold session has `helm chart` literally — grep wins on lexical exactness, hybrid matches but doesn't outperform.
|
||||
- The corpus is intentionally small for fast iteration. Hardening targets: paraphrased queries, synonym substitution, in-corpus distractors with shared keywords, longer multi-session chains.
|
||||
- Vector adapter not measured here — requires `OPENAI_API_KEY`; will be added in a follow-up scorecard alongside LongMemEval `_s`.
|
||||
@@ -0,0 +1,54 @@
|
||||
# <YYYY-MM-DD> — <benchmark-name>
|
||||
|
||||
**Commit:** `<sha>`
|
||||
**Bench:** LongMemEval `_s` / coding-agent-life-v1 / ...
|
||||
**N:** 500 / 15 / ...
|
||||
**K:** 5
|
||||
**Hardware:** macos-15 / ubuntu-22.04 / ...
|
||||
**OpenAI model:** text-embedding-3-small
|
||||
**Anthropic model:** N/A (no LLM in retrieval loop)
|
||||
|
||||
## Headline
|
||||
|
||||
agentmemory-hybrid: **R@5 = XX.XX%**, P@5 = XX.XX%, p50 latency = XXms
|
||||
|
||||
Beats grep baseline by +X.Xpt R@5, vector by +X.Xpt R@5.
|
||||
|
||||
## Per-adapter
|
||||
|
||||
| Adapter | P@5 | R@5 | Hit rate | p50 latency |
|
||||
|---|---|---|---|---|
|
||||
| grep | | | | |
|
||||
| vector | | | | |
|
||||
| agentmemory-hybrid | | | | |
|
||||
|
||||
## Per-question-type
|
||||
|
||||
| Type | grep R@5 | vector R@5 | agentmemory R@5 |
|
||||
|---|---|---|---|
|
||||
| single-session-bug | | | |
|
||||
| single-session-refactor | | | |
|
||||
| preference | | | |
|
||||
| multi-session-causal | | | |
|
||||
| temporal | | | |
|
||||
|
||||
## Methodology
|
||||
|
||||
- Sessions ingested via `POST /agentmemory/remember` with `type=eval-session`
|
||||
- Queries hit `POST /agentmemory/smart-search` with `limit=k*4`
|
||||
- No LLM in retrieval loop. Direct rank from hybrid scoring.
|
||||
- Ranks dedup by sessionId before truncating to K
|
||||
- Latency measured as init+query for LongMemEval (per-question fresh state), query-only for coding-life (shared state)
|
||||
|
||||
## Reproduce
|
||||
|
||||
```sh
|
||||
git checkout <sha>
|
||||
npm install --legacy-peer-deps
|
||||
OPENAI_API_KEY=sk-... AGENTMEMORY_BASE_URL=http://localhost:3111 \
|
||||
npm run eval:longmemeval -- --stratify 10
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
<what surprised, what regressed, what's load-bearing>
|
||||
Reference in New Issue
Block a user