chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,165 @@
|
||||
# AI Agent Memory: Benchmark Comparison
|
||||
|
||||
How agentmemory compares against other persistent memory solutions for AI coding agents.
|
||||
|
||||
All numbers here come from published benchmarks or public repositories. We link to primary sources wherever possible so you can reproduce.
|
||||
|
||||
---
|
||||
|
||||
## Retrieval Accuracy (LongMemEval)
|
||||
|
||||
[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) measures long-term memory retrieval across ~48 sessions per question on the S variant (500 questions, ~115K tokens each).
|
||||
|
||||
| System | Benchmark | R@5 | Notes |
|
||||
|---|---|---|---|
|
||||
| **agentmemory** (BM25 + Vector) | LongMemEval-S | **95.2%** | `all-MiniLM-L6-v2` embeddings, no API key |
|
||||
| agentmemory (BM25-only) | LongMemEval-S | 86.2% | Fallback when no embedding provider available |
|
||||
| MemPalace | LongMemEval-S | ~96.6% (self-reported) | Vendor-published number we have not independently reproduced. Vector-only with a larger embedding model and no agent-integration surface (no hooks, no MCP, no multi-agent) |
|
||||
| oracleagentmemory | LongMemEval | 94.4% (self-reported) | Vendor-published, scored with GPT-5.5 at "xhigh reasoning" and requires an Oracle AI Database. We have not reproduced it. agentmemory's 95.2% uses free local embeddings and no API key |
|
||||
| Letta / MemGPT | LoCoMo | 83.2% | Different benchmark (LoCoMo, not LongMemEval) |
|
||||
| Mem0 | LoCoMo | 68.5% | Different benchmark (LoCoMo, not LongMemEval) |
|
||||
|
||||
**⚠️ Apples vs oranges caveat:** only agentmemory's 95.2% is our own measured result, reproducible from the methodology below. Every other number here is the vendor's published claim, on a different benchmark or harness, that we have not independently reproduced: MemPalace and oracleagentmemory report LongMemEval (oracleagentmemory's run used GPT-5.5 at "xhigh reasoning" against an Oracle AI Database), while Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/). Treat them as ballpark vendor claims, not a head-to-head on identical data. We'd love to run every system on the same dataset; if any maintainer wants to collaborate, open an issue.
|
||||
|
||||
Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)
|
||||
|
||||
---
|
||||
|
||||
## Feature Matrix
|
||||
|
||||
| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | MemPalace | oracleagentmemory | Hippo |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | 54K+ | PyPI (Oracle) | Trending |
|
||||
| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | Memory API + app | Benchmark-focused OSS | Memory engine (Oracle DB) | Memory system |
|
||||
| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ❌ API-side extraction | ❌ Manual | ❌ API extraction | ❌ Manual |
|
||||
| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | Vector + RAG | Vector-only (large model) | Vector + semantic | Decay-weighted |
|
||||
| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | ❌ | Scoped only (user/agent/thread) | Multi-agent shared |
|
||||
| **Framework lock-in** | None | None | High | Standalone | None (drop-in wrappers) | None | Oracle Database | None |
|
||||
| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | Managed cloud | Vector store | Oracle AI Database | None |
|
||||
| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ❌ Cloud-only | ✅ | ✅ (needs Oracle DB) | ✅ |
|
||||
| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ✅ Auto-forget | ❌ | ❌ | ✅ Half-lives |
|
||||
| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | ❌ | ❌ | Episodic + semantic |
|
||||
| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ✅ Auto-resolve | ❌ | ❌ | ❌ |
|
||||
| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | Cloud dashboard | ❌ | ❌ | ❌ |
|
||||
| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | MCP + API | Standalone | Python API | Multi-agent shared |
|
||||
| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Python + TS | Python | Python only | Node |
|
||||
|
||||
---
|
||||
|
||||
## Token Efficiency
|
||||
|
||||
The main reason to use persistent memory at all: token cost. Here's what one year of heavy agent use looks like across approaches.
|
||||
|
||||
| Approach | Tokens / year | Cost / year | Notes |
|
||||
|---|---|---|---|
|
||||
| Paste full history into context | 19.5M+ | Impossible | Exceeds context window after ~200 observations |
|
||||
| LLM-summarized memory (extraction-based) | ~650K | ~$500 | Lossy — summarization drops detail |
|
||||
| **agentmemory (API embeddings)** | **~170K** | **~$10** | Token-budgeted, only relevant memories injected |
|
||||
| **agentmemory (local embeddings)** | **~170K** | **$0** | `all-MiniLM-L6-v2` runs in-process |
|
||||
| supermemory | Not published | Cloud pricing | Managed API, no local token budget |
|
||||
| Mem0 | Varies by integration | Varies | Extraction-based, no token budget |
|
||||
|
||||
**agentmemory ships with a built-in token savings calculator.** Run `npx @agentmemory/agentmemory status` after a few sessions and you'll see exactly how many tokens you've saved vs. pasting the full history.
|
||||
|
||||
---
|
||||
|
||||
## What Each Tool Is Best At
|
||||
|
||||
This isn't a "agentmemory wins everything" page. Different tools solve different problems.
|
||||
|
||||
**Choose agentmemory if you want:**
|
||||
- Automatic capture with zero manual `add()` calls
|
||||
- MCP server that works across Claude Code, Cursor, Codex, Gemini CLI, etc.
|
||||
- Hybrid BM25 + vector + graph search
|
||||
- Real-time viewer to see what your agent is learning
|
||||
- Self-hostable with zero external databases
|
||||
- Privacy filtering on API keys and secrets
|
||||
- Multi-agent coordination (leases, signals, routines)
|
||||
|
||||
**Choose Mem0 if you want:**
|
||||
- Framework-agnostic API to bolt onto an existing agent
|
||||
- Managed cloud option with a dashboard
|
||||
- Python + TypeScript SDKs for direct integration
|
||||
- Entity/relationship extraction as the primary abstraction
|
||||
|
||||
**Choose Letta/MemGPT if you want:**
|
||||
- A full agent runtime, not just memory
|
||||
- OS-inspired memory tiers (core/archival/recall)
|
||||
- Agents that self-edit their memory via function calls
|
||||
- Long-running conversational agents (weeks/months)
|
||||
|
||||
**Choose Khoj if you want:**
|
||||
- A personal AI second brain, not agent infrastructure
|
||||
- Document-first search over your files and the web
|
||||
- Obsidian/Notion/Emacs integrations
|
||||
- Scheduled automations and research tasks
|
||||
|
||||
**Choose supermemory if you want:**
|
||||
- A managed memory API with server-side auto-extraction and automatic forgetting
|
||||
- Drop-in wrappers for major AI frameworks (Vercel AI, LangChain, LangGraph)
|
||||
- A hosted dashboard with no infrastructure to run yourself
|
||||
- RAG plus memory served from a single query
|
||||
|
||||
**Choose MemPalace if you want:**
|
||||
- A simple, free, open-source vector memory store
|
||||
- To chase its self-reported retrieval benchmark (we have not reproduced it)
|
||||
- Pure retrieval over agent workflow features
|
||||
- Note: no auto-capture, no MCP, no multi-agent coordination, so you wire all integration yourself
|
||||
|
||||
**Choose oracleagentmemory if you want:**
|
||||
- You already run on Oracle AI Database and want memory inside it
|
||||
- Enterprise Oracle stack with vector search in the same database
|
||||
- LLM-backed extraction and are fine paying for a frontier model (their benchmark used GPT-5.5)
|
||||
- Note: Python-only, Oracle Database required, no MCP, no real-time viewer
|
||||
|
||||
**Choose Hippo if you want:**
|
||||
- Biologically-inspired memory model (decay, consolidation, sleep)
|
||||
- Multi-agent shared memory as a primary feature
|
||||
- "Forget by default, earn persistence through use" philosophy
|
||||
|
||||
---
|
||||
|
||||
## Running Your Own Benchmarks
|
||||
|
||||
We encourage you to measure this yourself rather than trust any README. Here's how:
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/rohitg00/agentmemory.git
|
||||
cd agentmemory && npm install
|
||||
|
||||
# Run LongMemEval-S
|
||||
npm run bench:longmemeval
|
||||
|
||||
# Run quality benchmark (240 observations, 20 queries)
|
||||
npm run bench:quality
|
||||
|
||||
# Run scale benchmark
|
||||
npm run bench:scale
|
||||
|
||||
# Run real embeddings benchmark
|
||||
npm run bench:real-embeddings
|
||||
```
|
||||
|
||||
Results land in `benchmark/results/`. All scripts, datasets, and results are committed for reproducibility.
|
||||
|
||||
---
|
||||
|
||||
## Corrections Welcome
|
||||
|
||||
If you maintain one of these tools and we got a number wrong, please open an issue or PR. We'd rather have accurate numbers than convenient ones.
|
||||
|
||||
If you want to add your tool to this comparison, open a PR with:
|
||||
1. A link to your benchmark methodology
|
||||
2. The metric and dataset you're measuring on
|
||||
3. A commit hash / version so we can reproduce
|
||||
|
||||
**Sources:**
|
||||
- Mem0 LoCoMo benchmark: [mem0.ai blog](https://mem0.ai)
|
||||
- Letta LoCoMo benchmark: [letta.com/blog/benchmarking-ai-agent-memory](https://letta.com/blog/benchmarking-ai-agent-memory)
|
||||
- LongMemEval paper: [arxiv.org/abs/2410.10813](https://arxiv.org/abs/2410.10813)
|
||||
- LoCoMo paper: [snap-stanford.github.io/LoCoMo](https://snap-stanford.github.io/LoCoMo/)
|
||||
@@ -0,0 +1,79 @@
|
||||
# LongMemEval-S Benchmark Results
|
||||
|
||||
[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is an academic benchmark for evaluating long-term memory in chat assistants. It tests 5 core abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
|
||||
|
||||
## Setup
|
||||
|
||||
- **Dataset**: LongMemEval-S (500 questions, ~48 sessions per question, ~115K tokens)
|
||||
- **Source**: [xiaowu0162/longmemeval-cleaned](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned)
|
||||
- **Metric**: `recall_any@K` — does ANY gold session appear in top-K retrieved results?
|
||||
- **Embedding model**: `all-MiniLM-L6-v2` (384 dimensions, local, no API key)
|
||||
- **No LLM in the loop**: Pure retrieval evaluation, no answer generation or judge
|
||||
|
||||
## Results
|
||||
|
||||
| System | R@5 | R@10 | R@20 | NDCG@10 | MRR |
|
||||
|---|---|---|---|---|---|
|
||||
| **agentmemory BM25+Vector** | **95.2%** | **98.6%** | **99.4%** | **87.9%** | **88.2%** |
|
||||
| agentmemory BM25-only | 86.2% | 94.6% | 98.6% | 73.0% | 71.5% |
|
||||
| MemPalace raw (vector-only) | 96.6% | ~97.6% | — | — | — |
|
||||
|
||||
### By Question Type (BM25+Vector)
|
||||
|
||||
| Type | R@5 | R@10 | Count |
|
||||
|---|---|---|---|
|
||||
| knowledge-update | 98.7% | 100.0% | 78 |
|
||||
| multi-session | 97.7% | 100.0% | 133 |
|
||||
| single-session-assistant | 96.4% | 98.2% | 56 |
|
||||
| temporal-reasoning | 95.5% | 97.7% | 133 |
|
||||
| single-session-user | 90.0% | 97.1% | 70 |
|
||||
| single-session-preference | 83.3% | 96.7% | 30 |
|
||||
|
||||
### By Question Type (BM25-only)
|
||||
|
||||
| Type | R@5 | R@10 | Count |
|
||||
|---|---|---|---|
|
||||
| knowledge-update | 92.3% | 98.7% | 78 |
|
||||
| single-session-user | 91.4% | 95.7% | 70 |
|
||||
| temporal-reasoning | 88.0% | 94.7% | 133 |
|
||||
| multi-session | 86.5% | 96.2% | 133 |
|
||||
| single-session-assistant | 80.4% | 91.1% | 56 |
|
||||
| single-session-preference | 60.0% | 80.0% | 30 |
|
||||
|
||||
## Analysis
|
||||
|
||||
1. **BM25+Vector (95.2%) nearly matches pure vector search (96.6%)** with only a 1.4pp gap. Both use the same embedding model (all-MiniLM-L6-v2).
|
||||
|
||||
2. **BM25 alone gets 86.2%** — keyword search with Porter stemming and synonym expansion is surprisingly effective on conversational data.
|
||||
|
||||
3. **Adding vectors to BM25 gives +9pp** (86.2% → 95.2%), the largest improvement from any single component.
|
||||
|
||||
4. **Preferences are the hardest category** for both BM25 (60%) and hybrid (83.3%). These require understanding implicit/indirect statements.
|
||||
|
||||
5. **Multi-session and knowledge-update are strongest** (97.7%+ hybrid). The hybrid approach excels when facts are distributed across sessions.
|
||||
|
||||
6. **R@10 reaches 98.6%** — nearly all gold sessions are found within the top 10 results.
|
||||
|
||||
## Important Notes on Methodology
|
||||
|
||||
- These are **retrieval recall** scores, not end-to-end QA accuracy. The official LongMemEval metric is QA accuracy (retrieve + generate answer + GPT-4o judge).
|
||||
- Systems on the actual LongMemEval QA leaderboard score 60-95% depending on the LLM reader (Oracle GPT-4o gets ~82.4%).
|
||||
- We do NOT claim these as "LongMemEval scores" — they are retrieval-only evaluations on the LongMemEval-S haystack.
|
||||
- Each question builds a fresh index from its ~48 sessions, searches with the question text, and checks if gold session IDs appear in results.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
```bash
|
||||
# Download dataset (264 MB)
|
||||
pip install huggingface_hub
|
||||
python3 -c "
|
||||
from huggingface_hub import hf_hub_download
|
||||
hf_hub_download(repo_id='xiaowu0162/longmemeval-cleaned', filename='longmemeval_s_cleaned.json', repo_type='dataset', local_dir='benchmark/data')
|
||||
"
|
||||
|
||||
# Run BM25-only
|
||||
npx tsx benchmark/longmemeval-bench.ts bm25
|
||||
|
||||
# Run BM25+Vector hybrid (requires @xenova/transformers)
|
||||
npx tsx benchmark/longmemeval-bench.ts hybrid
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
# agentmemory v0.6.0 — Search Quality Evaluation (Internal Dataset)
|
||||
|
||||
> For results on the academic LongMemEval-S benchmark (ICLR 2025, 500 questions), see [`LONGMEMEVAL.md`](LONGMEMEVAL.md) — **95.2% R@5, 98.6% R@10**.
|
||||
|
||||
**Date:** 2026-03-18T07:44:43.397Z
|
||||
**Dataset:** 240 synthetic observations across 30 sessions (internal coding project)
|
||||
**Queries:** 20 labeled queries with ground-truth relevance
|
||||
**Metric definitions:** Recall@K (fraction of relevant docs in top K), Precision@K (fraction of top K that are relevant), NDCG@10 (ranking quality), MRR (position of first relevant result)
|
||||
|
||||
## Head-to-Head Comparison
|
||||
|
||||
| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Latency | Tokens/query |
|
||||
|--------|----------|-----------|-------------|---------|-----|---------|--------------|
|
||||
| Built-in (CLAUDE.md / grep) | 37.0% | 55.8% | 78.0% | 80.3% | 82.5% | 0.50ms | 22,610 |
|
||||
| Built-in (200-line MEMORY.md) | 27.4% | 37.8% | 63.0% | 56.4% | 65.5% | 0.16ms | 7,938 |
|
||||
| BM25-only | 43.8% | 55.9% | 95.0% | 82.7% | 95.5% | 0.17ms | 3,142 |
|
||||
| Dual-stream (BM25+Vector) | 42.4% | 58.6% | 90.0% | 84.7% | 95.4% | 0.71ms | 3,142 |
|
||||
| Triple-stream (BM25+Vector+Graph) | 36.8% | 58.0% | 87.0% | 81.7% | 87.9% | 1.02ms | 3,142 |
|
||||
|
||||
## Why This Matters
|
||||
|
||||
**Recall improvement:** agentmemory triple-stream finds 58.0% of relevant memories at K=10 vs 55.8% for keyword grep (+4%)
|
||||
**Token savings:** agentmemory returns only the top 10 results (3,142 tokens) vs loading everything into context (22,610 tokens) — 86% reduction
|
||||
**200-line cap:** Claude Code's MEMORY.md is capped at 200 lines. With 240 observations, 37.8% recall at K=10 — memories from later sessions are simply invisible.
|
||||
|
||||
## Per-Query Breakdown (Triple-Stream)
|
||||
|
||||
| Query | Category | Recall@10 | NDCG@10 | MRR | Relevant | Latency |
|
||||
|-------|----------|-----------|---------|-----|----------|---------|
|
||||
| How did we set up authentication? | semantic | 50.0% | 100.0% | 100.0% | 20 | 1.7ms |
|
||||
| JWT token validation middleware | exact | 50.0% | 64.9% | 100.0% | 10 | 1.2ms |
|
||||
| PostgreSQL connection issues | semantic | 33.3% | 100.0% | 100.0% | 30 | 1.0ms |
|
||||
| Playwright test configuration | exact | 100.0% | 100.0% | 100.0% | 10 | 1.1ms |
|
||||
| Why did the production deployment fail? | cross-session | 33.3% | 100.0% | 100.0% | 30 | 0.8ms |
|
||||
| rate limiting implementation | exact | 80.0% | 64.1% | 33.3% | 10 | 0.7ms |
|
||||
| What security measures did we add? | semantic | 33.3% | 100.0% | 100.0% | 30 | 0.7ms |
|
||||
| database performance optimization | semantic | 0.0% | 0.0% | 7.1% | 25 | 0.8ms |
|
||||
| Kubernetes pod crash debugging | entity | 100.0% | 96.7% | 100.0% | 5 | 1.2ms |
|
||||
| Docker containerization setup | entity | 100.0% | 100.0% | 100.0% | 10 | 0.9ms |
|
||||
| How does caching work in the app? | semantic | 25.0% | 64.9% | 100.0% | 20 | 0.8ms |
|
||||
| test infrastructure and factories | exact | 50.0% | 64.9% | 100.0% | 10 | 0.7ms |
|
||||
| What happened with the OAuth callback error? | cross-session | 100.0% | 54.1% | 16.7% | 5 | 1.1ms |
|
||||
| monitoring and observability setup | semantic | 66.7% | 100.0% | 100.0% | 15 | 0.8ms |
|
||||
| Prisma ORM configuration | entity | 25.7% | 93.6% | 100.0% | 35 | 1.8ms |
|
||||
| CI/CD pipeline configuration | exact | 20.0% | 64.9% | 100.0% | 25 | 1.0ms |
|
||||
| memory leak debugging | cross-session | 100.0% | 100.0% | 100.0% | 5 | 0.7ms |
|
||||
| API design decisions | semantic | 25.0% | 64.9% | 100.0% | 20 | 1.4ms |
|
||||
| zod validation schemas | entity | 66.7% | 100.0% | 100.0% | 15 | 0.7ms |
|
||||
| infrastructure as code Terraform | entity | 100.0% | 100.0% | 100.0% | 5 | 1.5ms |
|
||||
|
||||
## By Query Category
|
||||
|
||||
| Category | Avg Recall@10 | Avg NDCG@10 | Avg MRR | Queries |
|
||||
|----------|---------------|-------------|---------|---------|
|
||||
| exact | 60.0% | 71.8% | 86.7% | 5 |
|
||||
| semantic | 33.3% | 75.7% | 86.7% | 7 |
|
||||
| cross-session | 77.8% | 84.7% | 72.2% | 3 |
|
||||
| entity | 78.5% | 98.1% | 100.0% | 5 |
|
||||
|
||||
## Context Window Analysis
|
||||
|
||||
The fundamental problem with built-in agent memory:
|
||||
|
||||
| Observations | MEMORY.md tokens | agentmemory tokens (top 10) | Savings | MEMORY.md reachable |
|
||||
|-------------|-----------------|---------------------------|---------|-------------------|
|
||||
| 240 | 12,000 | 3,142 | 74% | 83% |
|
||||
| 500 | 25,000 | 3,142 | 87% | 40% |
|
||||
| 1,000 | 50,000 | 3,142 | 94% | 20% |
|
||||
| 5,000 | 250,000 | 3,142 | 99% | 4% |
|
||||
|
||||
At 240 observations (our dataset), MEMORY.md already hits its 200-line cap and loses access to the most recent 40 observations. At 1,000 observations, 80% of memories are invisible. agentmemory always searches the full corpus.
|
||||
|
||||
---
|
||||
|
||||
*100 evaluations across 5 systems. Ground-truth labels assigned by concept matching against observation metadata.*
|
||||
@@ -0,0 +1,100 @@
|
||||
# benchmark/
|
||||
|
||||
Two kinds of numbers live in this directory:
|
||||
|
||||
1. **Quality / retrieval** — `longmemeval-bench.ts`, `quality-eval.ts`,
|
||||
`real-embeddings-eval.ts`, `scale-eval.ts`. Recall, precision, token
|
||||
savings. Documented in `LONGMEMEVAL.md`, `QUALITY.md`,
|
||||
`REAL-EMBEDDINGS.md`, `SCALE.md`.
|
||||
|
||||
2. **Load shape** — `load-100k.ts`. p50 / p90 / p99 latency and
|
||||
throughput against a running daemon. This is the file you want when
|
||||
somebody asks "what's p99 at 100k memories under concurrency 100?".
|
||||
|
||||
## load-100k.ts
|
||||
|
||||
Hand-rolled, dependency-free load harness. Issues real HTTP against a
|
||||
local agentmemory daemon at `http://localhost:3111`, records per-request
|
||||
latency with `performance.now()`, and writes a JSON report per run.
|
||||
|
||||
### What it measures
|
||||
|
||||
For each cell in the matrix `(N, concurrency, endpoint)` it records:
|
||||
|
||||
- `p50_ms`, `p90_ms`, `p99_ms` — nearest-rank percentiles.
|
||||
- `min_ms`, `max_ms`, `ops`, `errors`.
|
||||
- `throughput_per_sec` — wall-clock ops / sec for that cell.
|
||||
|
||||
Default matrix:
|
||||
|
||||
- `N` ∈ {1000, 10000, 100000} — number of memories seeded before the
|
||||
cell runs.
|
||||
- `C` ∈ {1, 10, 100} — concurrent in-flight requests during the cell.
|
||||
- Endpoints under test:
|
||||
- `POST /agentmemory/remember`
|
||||
- `POST /agentmemory/smart-search`
|
||||
- `GET /agentmemory/memories?latest=true`
|
||||
|
||||
Each cell issues `BENCH_OPS=200` requests by default — enough samples
|
||||
for stable p99 without dragging a 100k-seed run past tens of minutes.
|
||||
|
||||
### Why p99 is the number that matters
|
||||
|
||||
p50 tells you the median request feels fast. p90 tells you the bulk of
|
||||
requests feel fast. **p99 tells you the request your tail user hits when
|
||||
they really need it feels fast.** Capacity planning lives here — if you
|
||||
want to size a fleet, scale your daemon, or set an SLO, p99 is the
|
||||
number to plan against. p50 will lie to you.
|
||||
|
||||
### Running it
|
||||
|
||||
```bash
|
||||
# 1. Start the daemon however you normally do (npx, Docker, etc.)
|
||||
npx @agentmemory/agentmemory
|
||||
|
||||
# 2. From the repo root, in another shell:
|
||||
npm run bench:load
|
||||
```
|
||||
|
||||
To override the matrix:
|
||||
|
||||
```bash
|
||||
BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
|
||||
```
|
||||
|
||||
To have the harness spawn a daemon for the run (after `npm run build`):
|
||||
|
||||
```bash
|
||||
AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
|
||||
```
|
||||
|
||||
Other env knobs (see the file header for the canonical list):
|
||||
|
||||
- `AGENTMEMORY_URL` — base URL of the daemon (default
|
||||
`http://localhost:3111`).
|
||||
- `BENCH_SEED` — seed for the `mulberry32` content RNG. Same seed +
|
||||
same daemon build = byte-identical seed corpus.
|
||||
- `BENCH_OUT_DIR` — where the JSON report lands (default
|
||||
`benchmark/results/`).
|
||||
|
||||
### Where results land
|
||||
|
||||
`benchmark/results/load-100k-<short-git-sha>.json`. The harness
|
||||
`mkdir -p`s the directory. The file has a `schema_version: 1` field so
|
||||
future format changes don't silently break consumers.
|
||||
|
||||
### Content generation is seedable
|
||||
|
||||
Synthetic memory content is built from a small noun / verb / concept
|
||||
vocabulary fed by a `mulberry32(BENCH_SEED)` PRNG. Same seed + same
|
||||
build = same corpus. The point isn't "realistic" content (there isn't
|
||||
one realistic content); the point is **reproducibility** — re-running
|
||||
the harness against the same git sha should give the same content
|
||||
mixture going in, so latency variance comes from the daemon and not
|
||||
from JSON payload jitter.
|
||||
|
||||
### Publishing numbers per release
|
||||
|
||||
The release flow appends a `## Performance` section to `CHANGELOG.md`
|
||||
referencing the JSON in `benchmark/results/` for that release's git
|
||||
sha. p99 is the headline number; the JSON is the receipt.
|
||||
@@ -0,0 +1,67 @@
|
||||
# agentmemory v0.6.0 — Real Embeddings Quality Evaluation
|
||||
|
||||
**Date:** 2026-03-18T07:38:21.450Z
|
||||
**Platform:** darwin arm64, Node v20.20.0
|
||||
**Dataset:** 240 observations, 30 sessions, 20 labeled queries
|
||||
**Embedding model:** Xenova/all-MiniLM-L6-v2 (384d, local, no API key)
|
||||
|
||||
## Head-to-Head: Real Embeddings vs Keyword Search
|
||||
|
||||
| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Avg Latency | Tokens/query |
|
||||
|--------|----------|-----------|-------------|---------|-----|-------------|--------------|
|
||||
| Built-in (grep all) | 37.0% | 55.8% | 78.0% | 80.3% | 82.5% | 0.44ms | 19,462 |
|
||||
| BM25-only (stemmed+synonyms) | 43.8% | 55.9% | 95.0% | 82.7% | 95.5% | 0.26ms | 1,571 |
|
||||
| Dual-stream (BM25+Xenova) | 43.8% | 64.1% | 98.0% | 94.9% | 100.0% | 2.39ms | 1,571 |
|
||||
| Triple-stream (BM25+Xenova+Graph) | 43.8% | 64.1% | 98.0% | 94.9% | 100.0% | 2.07ms | 1,571 |
|
||||
|
||||
## Improvement from Real Embeddings
|
||||
|
||||
Adding real vector embeddings to BM25 improves recall@10 by **8.2 percentage points**.
|
||||
Token savings vs loading everything: **92%** (1,571 vs 19,462 tokens).
|
||||
|
||||
## Per-Query: Where Real Embeddings Win
|
||||
|
||||
Queries where dual-stream (real embeddings) outperforms BM25-only:
|
||||
|
||||
| Query | Category | BM25 Recall@10 | +Vector Recall@10 | Delta |
|
||||
|-------|----------|---------------|-------------------|-------|
|
||||
| How did we set up authentication? | semantic | 25.0% | 45.0% | +20.0pp ** |
|
||||
| Playwright test configuration | exact | 50.0% | 90.0% | +40.0pp ** |
|
||||
| database performance optimization | semantic | 0.0% | 40.0% | +40.0pp ** |
|
||||
| test infrastructure and factories | exact | 50.0% | 80.0% | +30.0pp ** |
|
||||
| Prisma ORM configuration | entity | 14.3% | 28.6% | +14.3pp ** |
|
||||
| CI/CD pipeline configuration | exact | 20.0% | 40.0% | +20.0pp ** |
|
||||
|
||||
## By Category Comparison
|
||||
|
||||
| Category | Built-in grep | BM25 (stemmed) | +Real Vectors | +Graph |
|
||||
|----------|--------------|----------------|--------------|--------|
|
||||
| exact | 48.0% | 54.0% | 72.0% | 72.0% |
|
||||
| semantic | 35.5% | 33.3% | 41.9% | 41.9% |
|
||||
| cross-session | 77.8% | 77.8% | 77.8% | 77.8% |
|
||||
| entity | 79.0% | 76.2% | 79.0% | 79.0% |
|
||||
|
||||
## Embedding Performance
|
||||
|
||||
| System | Embedding Time | Model | Dimensions |
|
||||
|--------|---------------|-------|------------|
|
||||
| Dual-stream (BM25+Xenova) | 3.1s | Xenova/all-MiniLM-L6-v2 | 384 |
|
||||
| Triple-stream (BM25+Xenova+Graph) | 2.9s | Xenova/all-MiniLM-L6-v2 | 384 |
|
||||
|
||||
Embedding is a one-time cost at ingestion. Search is sub-millisecond after indexing.
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. **Semantic queries improve most**: 8.6pp recall@10 gain from real embeddings
|
||||
2. **"database performance optimization"** — the hardest query — goes from BM25 0.0% to vector-augmented 40.0%
|
||||
3. **Entity/exact queries** are already well-served by BM25+stemming — vectors add marginal value
|
||||
4. **Local embeddings (Xenova)** run without API keys — zero cost, zero latency concerns
|
||||
|
||||
## Recommendation
|
||||
|
||||
Enable local embeddings by default (`EMBEDDING_PROVIDER=local` or install `@xenova/transformers`).
|
||||
This gives agentmemory genuine semantic search that built-in agent memories cannot match —
|
||||
understanding that "database performance optimization" relates to "N+1 query fix" and "eager loading".
|
||||
|
||||
---
|
||||
*All measurements use Xenova/all-MiniLM-L6-v2 local embeddings (384 dimensions, no API calls).*
|
||||
@@ -0,0 +1,110 @@
|
||||
# agentmemory v0.6.0 — Scale & Cross-Session Evaluation
|
||||
|
||||
**Date:** 2026-03-18T07:45:03.529Z
|
||||
**Platform:** darwin arm64, Node v20.20.0
|
||||
|
||||
## 1. Scale: agentmemory vs Built-in Memory
|
||||
|
||||
Every built-in agent memory (CLAUDE.md, .cursorrules, Cline's memory-bank) loads ALL memory into context every session. agentmemory searches and returns only relevant results.
|
||||
|
||||
| Observations | Sessions | Index Build | BM25 Search | Hybrid Search | Heap | Context Tokens (built-in) | Context Tokens (agentmemory) | Savings | Built-in Unreachable |
|
||||
|-------------|----------|------------|-------------|---------------|------|--------------------------|-----------------------------|---------|--------------------|
|
||||
| 240 | 30 | 177ms | 0.112ms | 0.63ms | 9MB | 10,504 | 1,924 | 82% | 17% |
|
||||
| 1,000 | 125 | 155ms | 0.317ms | 1.709ms | 6MB | 43,834 | 1,969 | 96% | 80% |
|
||||
| 5,000 | 625 | 810ms | 1.496ms | 8.58ms | 25MB | 220,335 | 1,972 | 99% | 96% |
|
||||
| 10,000 | 1250 | 1657ms | 3.195ms | 17.49ms | 1MB | 440,973 | 1,974 | 100% | 98% |
|
||||
| 50,000 | 6250 | 9182ms | 22.827ms | 108.722ms | 316MB | 2,216,173 | 1,981 | 100% | 100% |
|
||||
|
||||
### What the numbers mean
|
||||
|
||||
**Context Tokens (built-in):** How many tokens Claude Code/Cursor/Cline would consume loading ALL memory into the context window. At 5,000 observations, this is ~250K tokens — exceeding most context windows entirely.
|
||||
|
||||
**Context Tokens (agentmemory):** How many tokens the top-10 search results consume. Stays constant regardless of corpus size.
|
||||
|
||||
**Built-in Unreachable:** Percentage of memories that built-in systems CANNOT access because they exceed the 200-line MEMORY.md cap or context window limits. At 1,000 observations, 80% of your project history is invisible.
|
||||
|
||||
### Storage Costs
|
||||
|
||||
| Observations | BM25 Index | Vector Index (d=384) | Total Storage |
|
||||
|-------------|-----------|---------------------|---------------|
|
||||
| 240 | 395 KB | 494 KB | 0.9 MB |
|
||||
| 1,000 | 1,599 KB | 2,060 KB | 3.6 MB |
|
||||
| 5,000 | 8,006 KB | 10,298 KB | 17.9 MB |
|
||||
| 10,000 | 16,005 KB | 20,596 KB | 35.7 MB |
|
||||
| 50,000 | 80,126 KB | 102,979 KB | 178.8 MB |
|
||||
|
||||
## 2. Cross-Session Retrieval
|
||||
|
||||
Can the system find relevant information from past sessions? This is impossible for built-in memory once observations exceed the line/context cap.
|
||||
|
||||
| Query | Target Session | Gap | BM25 Found | BM25 Rank | Hybrid Found | Hybrid Rank | Built-in Visible |
|
||||
|-------|---------------|-----|-----------|-----------|-------------|-------------|-----------------|
|
||||
| How did we set up OAuth providers? | ses_005-009 | 24 | Yes | #1 | Yes | #1 | Yes |
|
||||
| What was the N+1 query fix? | ses_010-014 | 18 | Yes | #1 | Yes | #2 | Yes |
|
||||
| PostgreSQL full-text search setup | ses_010-014 | 17 | Yes | #1 | Yes | #1 | Yes |
|
||||
| bcrypt password hashing configuration | ses_005-009 | 20 | Yes | #1 | Yes | #1 | Yes |
|
||||
| Vitest unit testing setup | ses_020-024 | 9 | Yes | #1 | Yes | #1 | Yes |
|
||||
| webhook retry exponential backoff | ses_015-019 | 14 | Yes | #1 | Yes | #1 | Yes |
|
||||
| ESLint flat config migration | ses_000-004 | 29 | Yes | #1 | Yes | #1 | Yes |
|
||||
| Kubernetes HPA autoscaling configuration | ses_025-029 | 4 | Yes | #1 | Yes | #1 | No |
|
||||
| Prisma database seed script | ses_010-014 | 16 | Yes | #1 | Yes | #1 | Yes |
|
||||
| API cursor-based pagination | ses_015-019 | 14 | Yes | #1 | Yes | #1 | Yes |
|
||||
| CSRF protection double-submit cookie | ses_005-009 | 24 | Yes | #1 | Yes | #1 | Yes |
|
||||
| blue-green deployment rollback | ses_025-029 | 4 | Yes | #1 | Yes | #1 | No |
|
||||
|
||||
**Summary:** agentmemory BM25 found 12/12 cross-session queries. Hybrid found 12/12. Built-in memory (200-line cap) could only reach 10/12.
|
||||
|
||||
## 3. The Context Window Problem
|
||||
|
||||
```
|
||||
Agent context window: ~200K tokens
|
||||
System prompt + tools: ~20K tokens
|
||||
User conversation: ~30K tokens
|
||||
Available for memory: ~150K tokens
|
||||
|
||||
At 50 tokens/observation:
|
||||
200 observations = 10,000 tokens (fits, but 200-line cap hits first)
|
||||
1,000 observations = 50,000 tokens (33% of available budget)
|
||||
5,000 observations = 250,000 tokens (EXCEEDS total context window)
|
||||
|
||||
agentmemory top-10 results:
|
||||
Any corpus size = ~1,924 tokens (0.3% of budget)
|
||||
```
|
||||
|
||||
## 4. What Built-in Memory Cannot Do
|
||||
|
||||
| Capability | Built-in (CLAUDE.md) | agentmemory |
|
||||
|-----------|---------------------|-------------|
|
||||
| Semantic search | No (keyword grep only) | BM25 + vector + graph |
|
||||
| Scale beyond 200 lines | No (hard cap) | Unlimited |
|
||||
| Cross-session recall | Only if in 200-line window | Full corpus search |
|
||||
| Cross-agent sharing | No (per-agent files) | MCP + REST API |
|
||||
| Multi-agent coordination | No | Leases, signals, actions |
|
||||
| Temporal queries | No | Point-in-time graph |
|
||||
| Memory lifecycle | No (manual pruning) | Ebbinghaus decay + eviction |
|
||||
| Knowledge graph | No | Entity extraction + traversal |
|
||||
| Query expansion | No | LLM-generated reformulations |
|
||||
| Retention scoring | No | Time-frequency decay model |
|
||||
| Real-time dashboard | No (read files manually) | Viewer on :3113 |
|
||||
| Concurrent access | No (file lock) | Keyed mutex + KV store |
|
||||
|
||||
## 5. When to Use What
|
||||
|
||||
**Use built-in memory (CLAUDE.md) when:**
|
||||
- You have < 200 items to remember
|
||||
- Single agent, single project
|
||||
- Preferences and quick facts only
|
||||
- Zero setup is the priority
|
||||
|
||||
**Use agentmemory when:**
|
||||
- Project history exceeds 200 observations
|
||||
- You need to recall specific incidents from weeks ago
|
||||
- Multiple agents work on the same codebase
|
||||
- You want semantic search ("how does auth work?") not just keyword matching
|
||||
- You need to track memory quality, decay, and lifecycle
|
||||
- You want a shared memory layer across Claude Code, Cursor, Windsurf, etc.
|
||||
|
||||
Built-in memory is your sticky notes. agentmemory is the searchable database behind them.
|
||||
|
||||
---
|
||||
*Scale tests: 5 corpus sizes. Cross-session tests: 12 queries targeting specific past sessions.*
|
||||
@@ -0,0 +1,293 @@
|
||||
import type { CompressedObservation } from "../src/types.js";
|
||||
|
||||
export interface LabeledQuery {
|
||||
query: string;
|
||||
relevantObsIds: string[];
|
||||
description: string;
|
||||
category: "exact" | "semantic" | "temporal" | "cross-session" | "entity";
|
||||
}
|
||||
|
||||
const SESSION_COUNT = 30;
|
||||
const OBS_PER_SESSION = 8;
|
||||
|
||||
function ts(daysAgo: number): string {
|
||||
return new Date(Date.now() - daysAgo * 86400000).toISOString();
|
||||
}
|
||||
|
||||
const RAW_SESSIONS: Array<{
|
||||
sessionRange: [number, number];
|
||||
daysAgoRange: [number, number];
|
||||
project: string;
|
||||
observations: Array<Omit<CompressedObservation, "id" | "sessionId" | "timestamp">>;
|
||||
}> = [
|
||||
{
|
||||
sessionRange: [0, 4],
|
||||
daysAgoRange: [28, 25],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "command_run", title: "Initialize Next.js 15 project", subtitle: "create-next-app", facts: ["Created Next.js 15 app with App Router", "TypeScript template selected", "Tailwind CSS v4 configured"], narrative: "Initialized a new Next.js 15 project using create-next-app with TypeScript and Tailwind CSS. Selected the App Router layout.", concepts: ["nextjs", "typescript", "tailwind", "app-router"], files: ["package.json", "tsconfig.json", "tailwind.config.ts"], importance: 6 },
|
||||
{ type: "file_edit", title: "Configure ESLint with flat config", subtitle: "eslint.config.mjs", facts: ["Migrated to ESLint flat config format", "Added typescript-eslint plugin", "Configured import sorting rules"], narrative: "Set up ESLint using the new flat config format (eslint.config.mjs). Added typescript-eslint for type-aware linting and configured import sorting with eslint-plugin-import.", concepts: ["eslint", "linting", "code-quality", "typescript"], files: ["eslint.config.mjs", "package.json"], importance: 5 },
|
||||
{ type: "file_edit", title: "Set up Prettier with Tailwind plugin", subtitle: "Formatting", facts: ["Installed prettier and prettier-plugin-tailwindcss", "Added .prettierrc with semi: false, singleQuote: true", "Configured format-on-save in VS Code settings"], narrative: "Configured Prettier for automatic code formatting. Added the Tailwind CSS class sorting plugin. Set up VS Code to format on save.", concepts: ["prettier", "formatting", "tailwind", "developer-experience"], files: [".prettierrc", ".vscode/settings.json"], importance: 4 },
|
||||
{ type: "file_edit", title: "Create shared UI component library", subtitle: "Components", facts: ["Created Button, Input, Card, Badge components", "Used cva (class-variance-authority) for variant styling", "Added Radix UI primitives for accessibility"], narrative: "Built a shared component library with Button, Input, Card, and Badge components. Used class-variance-authority (cva) for type-safe variant styling and Radix UI primitives for keyboard navigation and screen reader support.", concepts: ["components", "ui-library", "radix-ui", "cva", "accessibility"], files: ["src/components/ui/button.tsx", "src/components/ui/input.tsx", "src/components/ui/card.tsx"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add global layout with navigation", subtitle: "Layout", facts: ["Created root layout with metadata", "Added responsive navigation bar", "Implemented mobile hamburger menu"], narrative: "Created the root layout component with SEO metadata, Open Graph tags, and a responsive navigation bar that collapses into a hamburger menu on mobile devices.", concepts: ["layout", "navigation", "responsive-design", "seo"], files: ["src/app/layout.tsx", "src/components/nav.tsx"], importance: 6 },
|
||||
{ type: "file_edit", title: "Configure path aliases and absolute imports", subtitle: "tsconfig", facts: ["Added @ alias pointing to src/", "Configured baseUrl for absolute imports"], narrative: "Set up TypeScript path aliases so imports can use @/components instead of relative paths. Configured baseUrl in tsconfig.json.", concepts: ["typescript", "path-aliases", "developer-experience"], files: ["tsconfig.json"], importance: 3 },
|
||||
{ type: "command_run", title: "Add Vitest for unit testing", subtitle: "Testing setup", facts: ["Installed vitest and @testing-library/react", "Created vitest.config.ts with jsdom environment", "Added test script to package.json"], narrative: "Set up Vitest as the unit testing framework with React Testing Library for component tests. Configured jsdom environment for DOM testing.", concepts: ["vitest", "testing", "react-testing-library", "configuration"], files: ["vitest.config.ts", "package.json"], importance: 5 },
|
||||
{ type: "file_edit", title: "Set up Husky pre-commit hooks", subtitle: "Git hooks", facts: ["Installed husky and lint-staged", "Pre-commit runs ESLint and Prettier", "Added commitlint for conventional commits"], narrative: "Configured Husky git hooks with lint-staged to run ESLint and Prettier on staged files before each commit. Added commitlint to enforce conventional commit message format.", concepts: ["husky", "git-hooks", "lint-staged", "commitlint", "ci"], files: [".husky/pre-commit", ".lintstagedrc", "commitlint.config.js"], importance: 4 },
|
||||
],
|
||||
},
|
||||
{
|
||||
sessionRange: [5, 9],
|
||||
daysAgoRange: [24, 20],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "file_edit", title: "Implement NextAuth.js v5 authentication", subtitle: "Auth setup", facts: ["Configured NextAuth.js v5 with Auth.js", "Added GitHub and Google OAuth providers", "Set up JWT session strategy with 30-day expiry"], narrative: "Implemented authentication using NextAuth.js v5 (Auth.js). Configured GitHub and Google as OAuth providers. Using JWT-based sessions with 30-day expiry instead of database sessions for simplicity.", concepts: ["nextauth", "authentication", "oauth", "jwt", "github", "google"], files: ["src/auth.ts", "src/app/api/auth/[...nextauth]/route.ts", ".env.local"], importance: 9 },
|
||||
{ type: "file_edit", title: "Create login and signup pages", subtitle: "Auth UI", facts: ["Built login page with OAuth buttons", "Added email/password form with validation", "Implemented error toast notifications"], narrative: "Created the login page with GitHub and Google OAuth sign-in buttons plus an email/password form. Used react-hook-form with zod validation. Added toast notifications for login errors.", concepts: ["login", "signup", "oauth", "form-validation", "react-hook-form", "zod"], files: ["src/app/login/page.tsx", "src/app/signup/page.tsx"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add middleware for route protection", subtitle: "Auth middleware", facts: ["Created middleware.ts to protect /dashboard routes", "Redirects unauthenticated users to /login", "Allows public access to /api/webhooks"], narrative: "Added Next.js middleware that checks for valid sessions on protected routes (/dashboard/*). Unauthenticated users are redirected to /login. The /api/webhooks path is excluded from auth checks for third-party integrations.", concepts: ["middleware", "route-protection", "authentication", "security"], files: ["src/middleware.ts"], importance: 8 },
|
||||
{ type: "file_edit", title: "Implement role-based access control", subtitle: "RBAC", facts: ["Added user roles: admin, editor, viewer", "Created withAuth HOC for role checking", "Stored roles in JWT custom claims"], narrative: "Implemented role-based access control with three roles: admin, editor, and viewer. Created a withAuth higher-order component that checks user roles before rendering protected components. Roles are stored as custom claims in the JWT token.", concepts: ["rbac", "authorization", "roles", "jwt-claims", "security"], files: ["src/lib/auth/rbac.ts", "src/lib/auth/with-auth.tsx"], importance: 8 },
|
||||
{ type: "file_edit", title: "Add password hashing with bcrypt", subtitle: "Security", facts: ["Using bcrypt with cost factor 12", "Added password strength validation (min 8 chars, mixed case, number)", "Implemented rate limiting on login endpoint (5 attempts per 15 min)"], narrative: "Added bcrypt password hashing with cost factor 12 for the email/password authentication flow. Implemented password strength validation requiring minimum 8 characters with mixed case and numbers. Added rate limiting on the login API endpoint: 5 attempts per 15-minute window per IP.", concepts: ["bcrypt", "password-hashing", "rate-limiting", "security", "validation"], files: ["src/lib/auth/password.ts", "src/app/api/auth/login/route.ts"], importance: 9 },
|
||||
{ type: "file_edit", title: "Create user profile settings page", subtitle: "User settings", facts: ["Profile page shows avatar, name, email", "Added avatar upload with S3 presigned URLs", "Implemented account deletion flow"], narrative: "Built the user profile settings page showing avatar, name, and email. Added avatar upload using S3 presigned URLs for direct browser-to-S3 uploads. Implemented a full account deletion flow with email confirmation.", concepts: ["user-profile", "settings", "s3", "file-upload", "account-deletion"], files: ["src/app/dashboard/settings/page.tsx", "src/app/api/upload/route.ts"], importance: 6 },
|
||||
{ type: "command_run", title: "Debug OAuth callback URL mismatch", subtitle: "Auth debugging", facts: ["GitHub OAuth callback failed with redirect_uri_mismatch", "Fixed: NEXTAUTH_URL was set to http:// but app served on https://", "Lesson: always use HTTPS in production OAuth callback URLs"], narrative: "Spent time debugging why GitHub OAuth login failed in production. The error was redirect_uri_mismatch. Root cause: NEXTAUTH_URL environment variable was set to http://localhost:3000 in production instead of the HTTPS production URL. Fixed by updating the environment variable.", concepts: ["oauth-debugging", "github", "callback-url", "environment-variables", "production"], files: [".env.production"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add CSRF protection to API routes", subtitle: "Security", facts: ["Implemented double-submit cookie pattern", "Added CSRF token generation in layout", "Validated CSRF token on all POST/PUT/DELETE requests"], narrative: "Added CSRF protection using the double-submit cookie pattern. A CSRF token is generated on page load and stored in both a cookie and a hidden form field. All mutating API requests (POST, PUT, DELETE) validate the token.", concepts: ["csrf", "security", "cookies", "api-protection"], files: ["src/lib/csrf.ts", "src/middleware.ts"], importance: 8 },
|
||||
],
|
||||
},
|
||||
{
|
||||
sessionRange: [10, 14],
|
||||
daysAgoRange: [19, 15],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "file_edit", title: "Set up Prisma ORM with PostgreSQL", subtitle: "Database", facts: ["Initialized Prisma with PostgreSQL provider", "Created User, Post, Comment, Tag models", "Generated migrations with prisma migrate dev"], narrative: "Set up Prisma ORM connecting to a PostgreSQL database. Defined the initial schema with User, Post, Comment, and Tag models including many-to-many relationships between Post and Tag.", concepts: ["prisma", "postgresql", "database", "orm", "schema", "migrations"], files: ["prisma/schema.prisma", "src/lib/db.ts"], importance: 9 },
|
||||
{ type: "file_edit", title: "Create database seed script", subtitle: "Seeding", facts: ["Created seed.ts with faker-generated data", "Seeds 10 users, 50 posts, 200 comments", "Runs via prisma db seed command"], narrative: "Built a database seed script using faker.js to generate realistic test data. Creates 10 users with posts, comments, and tags. Configured to run automatically on prisma db seed.", concepts: ["database", "seeding", "faker", "test-data", "prisma"], files: ["prisma/seed.ts", "package.json"], importance: 5 },
|
||||
{ type: "file_edit", title: "Implement server actions for CRUD operations", subtitle: "Data layer", facts: ["Created server actions for post CRUD", "Used Prisma transactions for multi-step operations", "Added revalidatePath after mutations"], narrative: "Implemented Next.js server actions for post create, read, update, and delete operations. Used Prisma transactions for operations that modify multiple tables. Called revalidatePath after mutations to refresh cached data.", concepts: ["server-actions", "crud", "prisma", "transactions", "revalidation", "caching"], files: ["src/app/actions/posts.ts"], importance: 8 },
|
||||
{ type: "command_run", title: "Fix N+1 query in post listing", subtitle: "Performance", facts: ["Identified N+1 query loading post authors individually", "Fixed with Prisma include for eager loading", "Query count dropped from 52 to 3"], narrative: "Discovered an N+1 query problem on the post listing page — each post was triggering a separate query to load its author. Fixed by using Prisma's include option for eager loading. Total query count dropped from 52 to 3.", concepts: ["n+1", "performance", "prisma", "eager-loading", "query-optimization"], files: ["src/app/actions/posts.ts"], importance: 8 },
|
||||
{ type: "file_edit", title: "Add full-text search with PostgreSQL tsvector", subtitle: "Search", facts: ["Created tsvector column on posts table", "Built GIN index for fast text search", "Implemented search API with ts_rank scoring"], narrative: "Added full-text search using PostgreSQL's built-in tsvector functionality. Created a generated tsvector column combining title and body, with a GIN index. The search API uses ts_rank for relevance scoring and supports phrase matching.", concepts: ["full-text-search", "postgresql", "tsvector", "gin-index", "search"], files: ["prisma/migrations/20260301_add_search.sql", "src/app/api/search/route.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Set up connection pooling with PgBouncer", subtitle: "Database infra", facts: ["Deployed PgBouncer in transaction pooling mode", "Configured max 25 client connections, 10 server connections", "Added DATABASE_URL_DIRECT for migrations (bypasses pooler)"], narrative: "Deployed PgBouncer as a connection pooler for PostgreSQL. Using transaction pooling mode to maximize connection reuse. Configured separate DATABASE_URL for application use (through pooler) and DATABASE_URL_DIRECT for migrations.", concepts: ["pgbouncer", "connection-pooling", "postgresql", "infrastructure"], files: ["docker-compose.yml", ".env"], importance: 7 },
|
||||
{ type: "command_run", title: "Debug Prisma migration drift", subtitle: "Database debugging", facts: ["prisma migrate deploy failed with drift detected", "Cause: manual SQL ALTER was run directly on production", "Resolution: ran prisma migrate resolve to mark migration as applied"], narrative: "Production deployment failed because Prisma detected schema drift — someone had run a manual ALTER TABLE directly on the production database. Resolved by using prisma migrate resolve to mark the conflicting migration as already applied.", concepts: ["prisma", "migration-drift", "database", "production", "debugging"], files: ["prisma/schema.prisma"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add Redis caching layer for expensive queries", subtitle: "Caching", facts: ["Used ioredis with 60-second TTL for post listings", "Implemented cache-aside pattern", "Added cache invalidation on post mutations"], narrative: "Added a Redis caching layer for expensive database queries. Post listings are cached for 60 seconds using a cache-aside pattern. Cache entries are invalidated when posts are created, updated, or deleted.", concepts: ["redis", "caching", "cache-aside", "ioredis", "performance"], files: ["src/lib/cache.ts", "src/app/actions/posts.ts"], importance: 7 },
|
||||
],
|
||||
},
|
||||
{
|
||||
sessionRange: [15, 19],
|
||||
daysAgoRange: [14, 10],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "file_edit", title: "Build REST API with input validation", subtitle: "API", facts: ["Created /api/v1/posts, /api/v1/users endpoints", "Used zod for request body validation", "Added consistent error response format with error codes"], narrative: "Built a versioned REST API under /api/v1/ with endpoints for posts and users. All request bodies are validated with zod schemas. Errors follow a consistent format with error codes, messages, and field-level details.", concepts: ["rest-api", "zod", "validation", "error-handling", "api-design"], files: ["src/app/api/v1/posts/route.ts", "src/app/api/v1/users/route.ts", "src/lib/api/errors.ts"], importance: 8 },
|
||||
{ type: "file_edit", title: "Implement cursor-based pagination", subtitle: "API pagination", facts: ["Replaced offset pagination with cursor-based approach", "Uses Prisma cursor with opaque base64-encoded cursors", "Returns hasNextPage and endCursor in response"], narrative: "Switched from offset-based to cursor-based pagination for the post listing API. Cursors are base64-encoded Prisma record IDs. Response includes hasNextPage boolean and endCursor for the client to request the next page.", concepts: ["pagination", "cursor-based", "prisma", "api-design", "performance"], files: ["src/app/api/v1/posts/route.ts", "src/lib/api/pagination.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add API rate limiting with Upstash Redis", subtitle: "Rate limiting", facts: ["Used @upstash/ratelimit with sliding window algorithm", "10 requests per 10 seconds per API key", "Returns X-RateLimit-Remaining header"], narrative: "Implemented API rate limiting using Upstash Redis with a sliding window algorithm. Each API key is limited to 10 requests per 10-second window. Rate limit status is communicated via standard X-RateLimit-* headers.", concepts: ["rate-limiting", "upstash", "redis", "api-security", "sliding-window"], files: ["src/middleware.ts", "src/lib/rate-limit.ts"], importance: 8 },
|
||||
{ type: "file_edit", title: "Create webhook system for external integrations", subtitle: "Webhooks", facts: ["Built webhook registration and delivery system", "Events: post.created, post.updated, user.signup", "Implemented retry with exponential backoff (max 3 retries)"], narrative: "Created a webhook system allowing external services to subscribe to events. Supports post.created, post.updated, and user.signup events. Webhook deliveries use exponential backoff with up to 3 retries on failure.", concepts: ["webhooks", "events", "integrations", "retry", "exponential-backoff"], files: ["src/lib/webhooks.ts", "src/app/api/v1/webhooks/route.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add OpenAPI specification with Swagger UI", subtitle: "API docs", facts: ["Generated OpenAPI 3.1 spec from zod schemas", "Added Swagger UI at /api/docs", "Included request/response examples"], narrative: "Generated an OpenAPI 3.1 specification from the existing zod validation schemas. Added Swagger UI accessible at /api/docs for interactive API documentation with request/response examples.", concepts: ["openapi", "swagger", "api-documentation", "zod"], files: ["src/app/api/docs/route.ts", "src/lib/openapi.ts"], importance: 5 },
|
||||
{ type: "command_run", title: "Debug 504 gateway timeout on large queries", subtitle: "Performance debugging", facts: ["Large post queries timing out after 30 seconds on Vercel", "Root cause: missing database index on posts.authorId", "Added composite index (authorId, createdAt DESC), query dropped to 50ms"], narrative: "Investigated 504 Gateway Timeout errors on the post listing endpoint in production (Vercel). Found that large queries filtering by author were doing a full table scan. Added a composite index on (authorId, createdAt DESC) which reduced query time from 30+ seconds to 50ms.", concepts: ["performance", "timeout", "database-index", "postgresql", "vercel", "debugging"], files: ["prisma/migrations/20260310_add_author_index.sql"], importance: 9 },
|
||||
{ type: "file_edit", title: "Implement API versioning strategy", subtitle: "API design", facts: ["URL-based versioning: /api/v1/, /api/v2/", "v1 deprecated with Sunset header", "Migration guide in API docs"], narrative: "Established an API versioning strategy using URL-based versioning (/api/v1/, /api/v2/). The v1 API returns a Sunset header indicating its deprecation date. Added a migration guide to the API documentation.", concepts: ["api-versioning", "deprecation", "sunset-header", "backward-compatibility"], files: ["src/app/api/v2/posts/route.ts", "src/lib/api/versioning.ts"], importance: 6 },
|
||||
{ type: "file_edit", title: "Add request logging with structured JSON", subtitle: "Observability", facts: ["Used pino for structured JSON logging", "Logs request method, path, status, duration, user ID", "Configured log levels per environment"], narrative: "Added structured JSON request logging using pino. Each request logs method, path, response status, duration in milliseconds, and authenticated user ID. Log levels are configured per environment (debug in dev, info in production).", concepts: ["logging", "pino", "observability", "structured-logging", "monitoring"], files: ["src/lib/logger.ts", "src/middleware.ts"], importance: 6 },
|
||||
],
|
||||
},
|
||||
{
|
||||
sessionRange: [20, 24],
|
||||
daysAgoRange: [9, 5],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "file_edit", title: "Write unit tests for auth module", subtitle: "Testing", facts: ["25 test cases covering login, signup, role checking", "Mocked Prisma client with vitest", "Achieved 92% coverage on auth module"], narrative: "Wrote comprehensive unit tests for the authentication module. 25 test cases covering login flow, signup validation, role-based access checks, and password hashing. Mocked the Prisma client using vitest's vi.mock. Achieved 92% code coverage.", concepts: ["unit-testing", "vitest", "mocking", "authentication", "coverage"], files: ["tests/unit/auth.test.ts", "tests/unit/rbac.test.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add E2E tests with Playwright", subtitle: "E2E testing", facts: ["Configured Playwright with Chrome and Firefox", "Tests: login flow, post CRUD, search, pagination", "Set up test database with Docker for isolation"], narrative: "Set up Playwright for end-to-end testing with Chrome and Firefox browsers. Created E2E tests for the complete login flow, post CRUD operations, search functionality, and pagination. Each test run gets a fresh database via Docker containers.", concepts: ["playwright", "e2e-testing", "docker", "test-isolation", "browser-testing"], files: ["playwright.config.ts", "tests/e2e/auth.spec.ts", "tests/e2e/posts.spec.ts", "docker-compose.test.yml"], importance: 8 },
|
||||
{ type: "command_run", title: "Fix flaky Playwright test on CI", subtitle: "CI debugging", facts: ["Test passed locally but failed in GitHub Actions", "Root cause: missing waitForNavigation after form submit", "Fixed by using page.waitForURL instead of waitForNavigation"], narrative: "Debugged a flaky Playwright test that passed locally but failed intermittently in GitHub Actions CI. The issue was a race condition after form submission — the test was checking the URL before navigation completed. Fixed by replacing the deprecated waitForNavigation with page.waitForURL.", concepts: ["playwright", "flaky-test", "ci", "github-actions", "debugging", "race-condition"], files: ["tests/e2e/auth.spec.ts"], importance: 6 },
|
||||
{ type: "file_edit", title: "Add API integration tests with supertest", subtitle: "API testing", facts: ["30 test cases for REST API endpoints", "Tests validation, auth, error responses, pagination", "Uses test database with transaction rollback"], narrative: "Created API integration tests using supertest. 30 test cases covering request validation, authentication requirements, error response formats, and cursor-based pagination. Each test runs in a database transaction that rolls back after completion.", concepts: ["integration-testing", "supertest", "api-testing", "transactions", "test-isolation"], files: ["tests/integration/api.test.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Set up test coverage reporting with codecov", subtitle: "Coverage", facts: ["Configured vitest coverage with v8 provider", "Minimum coverage thresholds: 80% branches, 85% lines", "Upload to Codecov in CI pipeline"], narrative: "Configured vitest code coverage using the v8 provider. Set minimum coverage thresholds at 80% for branches and 85% for lines. Coverage reports are uploaded to Codecov as part of the GitHub Actions CI pipeline.", concepts: ["code-coverage", "codecov", "vitest", "ci", "quality-gates"], files: ["vitest.config.ts", ".github/workflows/ci.yml"], importance: 5 },
|
||||
{ type: "file_edit", title: "Create test fixtures and factories", subtitle: "Test infrastructure", facts: ["Built factory functions for User, Post, Comment, Tag", "Uses faker for realistic data generation", "Supports partial overrides for specific test scenarios"], narrative: "Created test factory functions for all main models (User, Post, Comment, Tag). Factories use faker.js for realistic data and support partial overrides so individual tests can customize specific fields.", concepts: ["test-factories", "faker", "testing-infrastructure", "fixtures"], files: ["tests/fixtures/factories.ts"], importance: 5 },
|
||||
{ type: "command_run", title: "Debug memory leak in test suite", subtitle: "Test debugging", facts: ["Tests consuming 2GB+ RAM after 100+ test files", "Root cause: Prisma client not disconnected in afterAll", "Fixed by adding global teardown that calls prisma.$disconnect()"], narrative: "Investigated why the test suite was consuming over 2GB of RAM. The Prisma client was creating new connections in each test file but never disconnecting. Fixed by adding a global teardown hook that calls prisma.$disconnect().", concepts: ["memory-leak", "testing", "prisma", "debugging", "resource-management"], files: ["vitest.config.ts", "tests/setup.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add snapshot testing for API responses", subtitle: "Snapshot tests", facts: ["Added toMatchSnapshot for API response shapes", "Snapshot updates require --update flag", "Catches unintended breaking changes in API responses"], narrative: "Added snapshot testing for API response shapes to catch unintended breaking changes. Response bodies are compared against stored snapshots. Snapshots must be explicitly updated with the --update flag when intentional changes are made.", concepts: ["snapshot-testing", "api-testing", "regression-testing", "vitest"], files: ["tests/integration/api.test.ts", "tests/integration/__snapshots__/"], importance: 4 },
|
||||
],
|
||||
},
|
||||
{
|
||||
sessionRange: [25, 29],
|
||||
daysAgoRange: [4, 0],
|
||||
project: "webapp",
|
||||
observations: [
|
||||
{ type: "file_edit", title: "Create multi-stage Dockerfile", subtitle: "Docker", facts: ["Multi-stage build: deps → build → production", "Final image size 180MB (down from 1.2GB)", "Runs as non-root user with UID 1001"], narrative: "Created a multi-stage Dockerfile for the Next.js application. Stage 1 installs dependencies, stage 2 builds the app, stage 3 copies only production artifacts. Final image is 180MB (down from 1.2GB). Application runs as a non-root user for security.", concepts: ["docker", "multi-stage-build", "containerization", "security", "image-optimization"], files: ["Dockerfile", ".dockerignore"], importance: 7 },
|
||||
{ type: "file_edit", title: "Set up GitHub Actions CI/CD pipeline", subtitle: "CI/CD", facts: ["Matrix build: Node 18 and 20", "Jobs: lint, test, build, deploy", "Auto-deploy to Vercel on main branch push"], narrative: "Created a comprehensive GitHub Actions CI/CD pipeline with matrix builds for Node 18 and 20. Pipeline runs lint, test (with coverage), build, and deploy jobs. Merges to main automatically trigger Vercel deployment.", concepts: ["github-actions", "ci-cd", "deployment", "vercel", "automation"], files: [".github/workflows/ci.yml", ".github/workflows/deploy.yml"], importance: 8 },
|
||||
{ type: "file_edit", title: "Configure Kubernetes deployment manifests", subtitle: "K8s", facts: ["Created Deployment, Service, Ingress, HPA resources", "HPA: min 2, max 10 replicas, CPU target 70%", "Health checks: liveness on /healthz, readiness on /readyz"], narrative: "Created Kubernetes deployment manifests including Deployment, Service, Ingress, and HorizontalPodAutoscaler. HPA scales between 2 and 10 replicas targeting 70% CPU utilization. Added liveness and readiness probes for health monitoring.", concepts: ["kubernetes", "deployment", "hpa", "autoscaling", "health-checks", "ingress"], files: ["k8s/deployment.yaml", "k8s/service.yaml", "k8s/ingress.yaml", "k8s/hpa.yaml"], importance: 8 },
|
||||
{ type: "file_edit", title: "Add Terraform for AWS infrastructure", subtitle: "IaC", facts: ["VPC with public/private subnets across 3 AZs", "RDS PostgreSQL with Multi-AZ failover", "ElastiCache Redis cluster with 2 replicas"], narrative: "Created Terraform modules for AWS infrastructure. VPC spans 3 availability zones with public and private subnets. RDS PostgreSQL instance with Multi-AZ failover for high availability. ElastiCache Redis cluster with 2 read replicas.", concepts: ["terraform", "aws", "infrastructure-as-code", "vpc", "rds", "elasticache"], files: ["terraform/main.tf", "terraform/vpc.tf", "terraform/rds.tf", "terraform/redis.tf"], importance: 8 },
|
||||
{ type: "command_run", title: "Debug Kubernetes pod crash loop", subtitle: "K8s debugging", facts: ["Pods in CrashLoopBackOff status", "Root cause: DATABASE_URL secret not mounted correctly", "Fixed: Secret key name was 'database-url' but env var expected 'DATABASE_URL'"], narrative: "Debugged pods stuck in CrashLoopBackOff. The application was failing to start because the DATABASE_URL environment variable was empty. Root cause: the Kubernetes secret had the key 'database-url' (kebab-case) but the secretKeyRef expected 'DATABASE_URL' (uppercase).", concepts: ["kubernetes", "debugging", "crashloopbackoff", "secrets", "environment-variables"], files: ["k8s/deployment.yaml", "k8s/secrets.yaml"], importance: 8 },
|
||||
{ type: "file_edit", title: "Set up Datadog monitoring and alerting", subtitle: "Monitoring", facts: ["Deployed Datadog agent as DaemonSet", "Custom metrics: request latency, error rate, DB query time", "Alerts: p99 latency > 500ms, error rate > 1%"], narrative: "Deployed the Datadog monitoring agent as a Kubernetes DaemonSet. Created custom metrics for request latency, error rate, and database query time. Set up alerts that trigger when p99 latency exceeds 500ms or error rate exceeds 1%.", concepts: ["datadog", "monitoring", "alerting", "observability", "kubernetes"], files: ["k8s/datadog-agent.yaml", "src/lib/metrics.ts"], importance: 7 },
|
||||
{ type: "file_edit", title: "Implement blue-green deployment strategy", subtitle: "Deployment", facts: ["Two identical environments: blue and green", "Health check must pass before traffic switch", "Instant rollback by switching back to previous color"], narrative: "Implemented blue-green deployment strategy. Two identical environments run simultaneously — deploy to the inactive one, run health checks, then switch traffic via Kubernetes service selector update. Rollback is instant by pointing traffic back to the previous color.", concepts: ["blue-green", "deployment-strategy", "zero-downtime", "rollback", "kubernetes"], files: ["k8s/blue-deployment.yaml", "k8s/green-deployment.yaml", "scripts/deploy.sh"], importance: 7 },
|
||||
{ type: "file_edit", title: "Add Prometheus metrics and Grafana dashboards", subtitle: "Observability", facts: ["Exported custom metrics via /metrics endpoint", "Metrics: http_request_duration, db_query_duration, cache_hit_ratio", "Created Grafana dashboard with request rate, latency, error panels"], narrative: "Added Prometheus metrics export on a /metrics endpoint. Custom metrics include HTTP request duration histogram, database query duration, and cache hit ratio. Created a Grafana dashboard with panels for request rate, latency percentiles, error rate, and cache performance.", concepts: ["prometheus", "grafana", "metrics", "observability", "dashboards"], files: ["src/lib/metrics.ts", "grafana/dashboard.json"], importance: 6 },
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
export function generateDataset(): {
|
||||
observations: CompressedObservation[];
|
||||
queries: LabeledQuery[];
|
||||
sessions: Map<string, string[]>;
|
||||
} {
|
||||
const observations: CompressedObservation[] = [];
|
||||
const sessions = new Map<string, string[]>();
|
||||
|
||||
for (const group of RAW_SESSIONS) {
|
||||
const [sStart, sEnd] = group.sessionRange;
|
||||
const [dStart, dEnd] = group.daysAgoRange;
|
||||
|
||||
for (let s = sStart; s <= sEnd; s++) {
|
||||
const sessionId = `ses_${s.toString().padStart(3, "0")}`;
|
||||
const daysAgo = dStart - ((s - sStart) / Math.max(1, sEnd - sStart)) * (dStart - dEnd);
|
||||
const obsIds: string[] = [];
|
||||
|
||||
const obsPerSession = Math.min(group.observations.length, OBS_PER_SESSION);
|
||||
for (let o = 0; o < obsPerSession; o++) {
|
||||
const idx = ((s - sStart) * obsPerSession + o) % group.observations.length;
|
||||
const raw = group.observations[idx];
|
||||
const obsId = `obs_${sessionId}_${o.toString().padStart(2, "0")}`;
|
||||
const hourOffset = o * 0.5;
|
||||
|
||||
observations.push({
|
||||
id: obsId,
|
||||
sessionId,
|
||||
timestamp: ts(daysAgo - hourOffset / 24),
|
||||
...raw,
|
||||
});
|
||||
obsIds.push(obsId);
|
||||
}
|
||||
sessions.set(sessionId, obsIds);
|
||||
}
|
||||
}
|
||||
|
||||
const queries: LabeledQuery[] = [
|
||||
{
|
||||
query: "How did we set up authentication?",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["nextauth", "authentication", "oauth", "jwt", "login", "signup"].includes(c))).map(o => o.id),
|
||||
description: "Should find all auth-related observations across sessions 5-9",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "JWT token validation middleware",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("jwt") || (o.concepts.includes("middleware") && o.concepts.includes("authentication"))).map(o => o.id),
|
||||
description: "Exact match on JWT middleware setup",
|
||||
category: "exact",
|
||||
},
|
||||
{
|
||||
query: "PostgreSQL connection issues",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["postgresql", "pgbouncer", "connection-pooling", "database"].includes(c))).map(o => o.id),
|
||||
description: "Should find database connection and pooling observations",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "Playwright test configuration",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("playwright") || (o.concepts.includes("e2e-testing"))).map(o => o.id),
|
||||
description: "E2E testing setup with Playwright",
|
||||
category: "exact",
|
||||
},
|
||||
{
|
||||
query: "Why did the production deployment fail?",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["debugging", "production", "crashloopbackoff", "timeout", "migration-drift"].includes(c))).map(o => o.id),
|
||||
description: "Cross-session: find all production debugging incidents",
|
||||
category: "cross-session",
|
||||
},
|
||||
{
|
||||
query: "rate limiting implementation",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("rate-limiting")).map(o => o.id),
|
||||
description: "Rate limiting across auth and API modules",
|
||||
category: "exact",
|
||||
},
|
||||
{
|
||||
query: "What security measures did we add?",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["security", "csrf", "bcrypt", "rate-limiting", "rbac", "password-hashing"].includes(c))).map(o => o.id),
|
||||
description: "Broad semantic: all security-related work",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "database performance optimization",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["n+1", "query-optimization", "database-index", "performance", "eager-loading", "caching"].includes(c))).map(o => o.id),
|
||||
description: "Performance optimizations across database and caching",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "Kubernetes pod crash debugging",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["crashloopbackoff", "kubernetes"].includes(c)) && o.concepts.includes("debugging")).map(o => o.id),
|
||||
description: "Specific K8s debugging incident",
|
||||
category: "entity",
|
||||
},
|
||||
{
|
||||
query: "Docker containerization setup",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["docker", "multi-stage-build", "containerization", "dockerfile"].includes(c))).map(o => o.id),
|
||||
description: "Docker-related observations",
|
||||
category: "entity",
|
||||
},
|
||||
{
|
||||
query: "How does caching work in the app?",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["redis", "caching", "cache-aside", "ioredis", "elasticache"].includes(c))).map(o => o.id),
|
||||
description: "All caching-related observations",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "test infrastructure and factories",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["test-factories", "testing-infrastructure", "fixtures", "mocking"].includes(c))).map(o => o.id),
|
||||
description: "Test setup infrastructure",
|
||||
category: "exact",
|
||||
},
|
||||
{
|
||||
query: "What happened with the OAuth callback error?",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["oauth-debugging", "callback-url"].includes(c))).map(o => o.id),
|
||||
description: "Specific debugging incident recall",
|
||||
category: "cross-session",
|
||||
},
|
||||
{
|
||||
query: "monitoring and observability setup",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["datadog", "prometheus", "grafana", "monitoring", "observability", "alerting", "metrics", "logging", "pino"].includes(c))).map(o => o.id),
|
||||
description: "All monitoring/observability observations",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "Prisma ORM configuration",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("prisma")).map(o => o.id),
|
||||
description: "All Prisma-related observations",
|
||||
category: "entity",
|
||||
},
|
||||
{
|
||||
query: "CI/CD pipeline configuration",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["ci-cd", "github-actions", "deployment", "ci"].includes(c))).map(o => o.id),
|
||||
description: "CI/CD related observations",
|
||||
category: "exact",
|
||||
},
|
||||
{
|
||||
query: "memory leak debugging",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("memory-leak")).map(o => o.id),
|
||||
description: "Memory leak incidents (WebSocket handler, test suite)",
|
||||
category: "cross-session",
|
||||
},
|
||||
{
|
||||
query: "API design decisions",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["rest-api", "api-design", "api-versioning", "pagination", "openapi", "error-handling"].includes(c))).map(o => o.id),
|
||||
description: "API design and architecture decisions",
|
||||
category: "semantic",
|
||||
},
|
||||
{
|
||||
query: "zod validation schemas",
|
||||
relevantObsIds: observations.filter(o => o.concepts.includes("zod")).map(o => o.id),
|
||||
description: "Where zod is used for validation",
|
||||
category: "entity",
|
||||
},
|
||||
{
|
||||
query: "infrastructure as code Terraform",
|
||||
relevantObsIds: observations.filter(o => o.concepts.some(c => ["terraform", "infrastructure-as-code", "aws", "vpc", "rds", "elasticache"].includes(c))).map(o => o.id),
|
||||
description: "Terraform/IaC observations",
|
||||
category: "entity",
|
||||
},
|
||||
];
|
||||
|
||||
return { observations, queries, sessions };
|
||||
}
|
||||
|
||||
export function generateScaleDataset(count: number): CompressedObservation[] {
|
||||
const base = generateDataset().observations;
|
||||
const result: CompressedObservation[] = [];
|
||||
|
||||
for (let i = 0; i < count; i++) {
|
||||
const src = base[i % base.length];
|
||||
result.push({
|
||||
...src,
|
||||
id: `obs_scale_${i.toString().padStart(6, "0")}`,
|
||||
sessionId: `ses_${Math.floor(i / 8).toString().padStart(4, "0")}`,
|
||||
timestamp: ts(Math.random() * 90),
|
||||
title: `${src.title} (iteration ${i})`,
|
||||
narrative: `${src.narrative} [Scale test variant ${i}, session group ${Math.floor(i / 8)}]`,
|
||||
});
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
/**
|
||||
* Nearest-rank percentile over a pre-sorted ascending array of numbers.
|
||||
*
|
||||
* No dependencies, no allocation. The caller is responsible for sorting
|
||||
* the input ascending (`arr.sort((a, b) => a - b)`) — sorting in here
|
||||
* would hide an O(n log n) cost in what looks like a cheap lookup.
|
||||
*
|
||||
* @param sorted Ascending-sorted samples. Empty array returns `NaN`.
|
||||
* @param p Percentile in [0, 100]. Values outside the range are clamped.
|
||||
* @returns The sample at the nearest rank, or `NaN` for empty input.
|
||||
*/
|
||||
export function pXX(sorted: number[], p: number): number {
|
||||
const n = sorted.length;
|
||||
if (n === 0) return NaN;
|
||||
const clamped = Math.max(0, Math.min(100, p));
|
||||
if (clamped === 0) return sorted[0]!;
|
||||
if (clamped === 100) return sorted[n - 1]!;
|
||||
// Nearest-rank: rank = ceil(p/100 * n), index = rank - 1.
|
||||
const rank = Math.ceil((clamped / 100) * n);
|
||||
const idx = Math.min(n - 1, Math.max(0, rank - 1));
|
||||
return sorted[idx]!;
|
||||
}
|
||||
@@ -0,0 +1,528 @@
|
||||
/**
|
||||
* Load harness — seeds N synthetic memories against a local agentmemory
|
||||
* daemon, then drives a matrix of (N, concurrency, endpoint) cells and
|
||||
* records p50 / p90 / p99 latency + throughput per cell.
|
||||
*
|
||||
* Spec: GitHub issue #346.
|
||||
*
|
||||
* Runs against an already-running daemon at `http://localhost:3111` by
|
||||
* default. Set `AGENTMEMORY_BENCH_AUTOSTART=1` to spawn one via
|
||||
* `node dist/cli.js start` for the duration of the run.
|
||||
*
|
||||
* Env knobs:
|
||||
* AGENTMEMORY_BENCH_AUTOSTART "1" to spawn the daemon (default: assume up)
|
||||
* AGENTMEMORY_URL base URL of the daemon (default: http://localhost:3111)
|
||||
* BENCH_N comma-separated N sizes (default: 1000,10000,100000)
|
||||
* BENCH_C comma-separated concurrency levels (default: 1,10,100)
|
||||
* BENCH_OPS ops per cell during measurement (default: 200)
|
||||
* BENCH_SEED seed for the mulberry32 RNG (default: 0xC0FFEE)
|
||||
* BENCH_OUT_DIR results dir (default: benchmark/results)
|
||||
*
|
||||
* The harness writes one JSON file per run named
|
||||
* `load-100k-<short-git-sha>.json`. The git sha is best-effort — falls
|
||||
* back to a timestamp when run outside a checkout.
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, writeFileSync, existsSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
|
||||
import { pXX } from "./lib/percentiles.js";
|
||||
|
||||
/** Seedable PRNG. Mulberry32 — 32-bit state, uniform output in [0, 1). */
|
||||
function mulberry32(seed: number): () => number {
|
||||
let s = seed >>> 0;
|
||||
return () => {
|
||||
s = (s + 0x6d2b79f5) >>> 0;
|
||||
let t = s;
|
||||
t = Math.imul(t ^ (t >>> 15), t | 1);
|
||||
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
const NOUNS = [
|
||||
"cache", "queue", "router", "stream", "shard", "lock", "buffer", "worker",
|
||||
"engine", "trigger", "function", "memory", "index", "graph", "vector",
|
||||
"session", "observation", "summary", "embedding", "tokenizer", "scheduler",
|
||||
"consumer", "producer", "channel", "actor", "pipeline", "watcher", "pool",
|
||||
];
|
||||
const VERBS = [
|
||||
"flushes", "rotates", "compacts", "rebalances", "drains", "warms",
|
||||
"expires", "deduplicates", "snapshots", "replays", "promotes", "demotes",
|
||||
"merges", "splits", "indexes", "scans", "compresses", "uploads",
|
||||
];
|
||||
const CONCEPTS = [
|
||||
"throughput", "latency", "backpressure", "consistency", "isolation",
|
||||
"durability", "idempotency", "fan-out", "cardinality", "skew",
|
||||
"hot-path", "cold-start", "tail-latency", "saturation", "quiescence",
|
||||
];
|
||||
|
||||
function buildContent(rng: () => number, i: number): string {
|
||||
const n = NOUNS[Math.floor(rng() * NOUNS.length)]!;
|
||||
const v = VERBS[Math.floor(rng() * VERBS.length)]!;
|
||||
const c1 = CONCEPTS[Math.floor(rng() * CONCEPTS.length)]!;
|
||||
const c2 = CONCEPTS[Math.floor(rng() * CONCEPTS.length)]!;
|
||||
const k = Math.floor(rng() * 9999);
|
||||
return `seed-${i} the ${n} ${v} ${c1} under ${c2} pressure (k=${k})`;
|
||||
}
|
||||
|
||||
interface RunConfig {
|
||||
baseUrl: string;
|
||||
Ns: number[];
|
||||
Cs: number[];
|
||||
opsPerCell: number;
|
||||
seed: number;
|
||||
outDir: string;
|
||||
autoStart: boolean;
|
||||
}
|
||||
|
||||
function parseIntList(raw: string | undefined, fallback: number[]): number[] {
|
||||
if (!raw) return fallback;
|
||||
const out = raw
|
||||
.split(",")
|
||||
.map((s) => parseInt(s.trim(), 10))
|
||||
.filter((n) => Number.isFinite(n) && n > 0);
|
||||
return out.length > 0 ? out : fallback;
|
||||
}
|
||||
|
||||
function loadConfig(): RunConfig {
|
||||
return {
|
||||
baseUrl: (process.env["AGENTMEMORY_URL"] || "http://localhost:3111").replace(
|
||||
/\/+$/,
|
||||
"",
|
||||
),
|
||||
Ns: parseIntList(process.env["BENCH_N"], [1000, 10000, 100000]),
|
||||
Cs: parseIntList(process.env["BENCH_C"], [1, 10, 100]),
|
||||
opsPerCell: parseInt(process.env["BENCH_OPS"] || "200", 10) || 200,
|
||||
seed: parseInt(process.env["BENCH_SEED"] || "12648430", 10) || 12648430,
|
||||
outDir:
|
||||
process.env["BENCH_OUT_DIR"] ||
|
||||
resolve(process.cwd(), "benchmark", "results"),
|
||||
autoStart: process.env["AGENTMEMORY_BENCH_AUTOSTART"] === "1",
|
||||
};
|
||||
}
|
||||
|
||||
async function waitForLivez(baseUrl: string, timeoutMs: number): Promise<void> {
|
||||
const start = Date.now();
|
||||
let lastErr: unknown = null;
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const res = await fetch(`${baseUrl}/agentmemory/livez`, {
|
||||
signal: AbortSignal.timeout(2000),
|
||||
});
|
||||
if (res.ok) return;
|
||||
lastErr = new Error(`livez HTTP ${res.status}`);
|
||||
} catch (err) {
|
||||
lastErr = err;
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
}
|
||||
const reason =
|
||||
lastErr instanceof Error ? lastErr.message : String(lastErr ?? "unknown");
|
||||
throw new Error(
|
||||
`daemon at ${baseUrl} did not become ready within ${timeoutMs} ms: ${reason}`,
|
||||
);
|
||||
}
|
||||
|
||||
function maybeStartDaemon(): ChildProcess | null {
|
||||
const candidates = ["dist/cli.mjs", "dist/cli.js"].map((p) =>
|
||||
resolve(process.cwd(), p),
|
||||
);
|
||||
const cliPath = candidates.find((p) => existsSync(p));
|
||||
if (!cliPath) {
|
||||
throw new Error(
|
||||
`AGENTMEMORY_BENCH_AUTOSTART=1 but neither ${candidates.join(" nor ")} exists — run \`npm run build\` first`,
|
||||
);
|
||||
}
|
||||
const child = spawn(process.execPath, [cliPath, "start"], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
detached: false,
|
||||
});
|
||||
child.stdout?.on("data", () => {
|
||||
/* drain */
|
||||
});
|
||||
child.stderr?.on("data", () => {
|
||||
/* drain */
|
||||
});
|
||||
return child;
|
||||
}
|
||||
|
||||
function shortGitSha(): string {
|
||||
try {
|
||||
const sha = execFileSync("git", ["rev-parse", "--short", "HEAD"], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
}).trim();
|
||||
if (sha) return sha;
|
||||
} catch {
|
||||
/* no git */
|
||||
}
|
||||
return `nogit-${Date.now().toString(36)}`;
|
||||
}
|
||||
|
||||
interface CellResult {
|
||||
endpoint: string;
|
||||
N: number;
|
||||
C: number;
|
||||
ops: number;
|
||||
errors: number;
|
||||
wall_ms: number;
|
||||
throughput_per_sec: number;
|
||||
p50_ms: number;
|
||||
p90_ms: number;
|
||||
p99_ms: number;
|
||||
min_ms: number;
|
||||
max_ms: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue `total` requests against `fetcher`, capped at `concurrency`
|
||||
* in-flight at any moment. Records per-request latency in ms.
|
||||
*/
|
||||
async function driveLoad(
|
||||
concurrency: number,
|
||||
total: number,
|
||||
fetcher: (i: number) => Promise<void>,
|
||||
): Promise<{ latencies: number[]; errors: number; wallMs: number }> {
|
||||
const latencies: number[] = [];
|
||||
let errors = 0;
|
||||
let issued = 0;
|
||||
const wallStart = performance.now();
|
||||
|
||||
async function worker(): Promise<void> {
|
||||
while (true) {
|
||||
const i = issued++;
|
||||
if (i >= total) return;
|
||||
const t0 = performance.now();
|
||||
try {
|
||||
await fetcher(i);
|
||||
latencies.push(performance.now() - t0);
|
||||
} catch {
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const workers = Array.from({ length: Math.max(1, concurrency) }, () =>
|
||||
worker(),
|
||||
);
|
||||
await Promise.allSettled(workers);
|
||||
const wallMs = performance.now() - wallStart;
|
||||
return { latencies, errors, wallMs };
|
||||
}
|
||||
|
||||
function summarize(
|
||||
endpoint: string,
|
||||
N: number,
|
||||
C: number,
|
||||
latencies: number[],
|
||||
errors: number,
|
||||
wallMs: number,
|
||||
): CellResult {
|
||||
const sorted = latencies.slice().sort((a, b) => a - b);
|
||||
const ops = sorted.length;
|
||||
return {
|
||||
endpoint,
|
||||
N,
|
||||
C,
|
||||
ops,
|
||||
errors,
|
||||
wall_ms: Math.round(wallMs * 1000) / 1000,
|
||||
throughput_per_sec:
|
||||
wallMs > 0 ? Math.round((ops / (wallMs / 1000)) * 100) / 100 : 0,
|
||||
p50_ms: Math.round(pXX(sorted, 50) * 1000) / 1000,
|
||||
p90_ms: Math.round(pXX(sorted, 90) * 1000) / 1000,
|
||||
p99_ms: Math.round(pXX(sorted, 99) * 1000) / 1000,
|
||||
min_ms: ops > 0 ? Math.round(sorted[0]! * 1000) / 1000 : NaN,
|
||||
max_ms: ops > 0 ? Math.round(sorted[ops - 1]! * 1000) / 1000 : NaN,
|
||||
};
|
||||
}
|
||||
|
||||
async function seedMemories(
|
||||
baseUrl: string,
|
||||
count: number,
|
||||
rng: () => number,
|
||||
seedConcurrency = 32,
|
||||
): Promise<{ seeded: number; errors: number; wallMs: number }> {
|
||||
let issued = 0;
|
||||
let seeded = 0;
|
||||
let errors = 0;
|
||||
const t0 = performance.now();
|
||||
async function worker(): Promise<void> {
|
||||
while (true) {
|
||||
const i = issued++;
|
||||
if (i >= count) return;
|
||||
const body = JSON.stringify({
|
||||
content: buildContent(rng, i),
|
||||
type: "observation",
|
||||
});
|
||||
try {
|
||||
const res = await fetch(`${baseUrl}/agentmemory/remember`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
if (res.ok) {
|
||||
seeded++;
|
||||
} else {
|
||||
errors++;
|
||||
}
|
||||
// drain body to free the socket
|
||||
await res.text().catch(() => "");
|
||||
} catch {
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
}
|
||||
await Promise.allSettled(
|
||||
Array.from({ length: seedConcurrency }, () => worker()),
|
||||
);
|
||||
return { seeded, errors, wallMs: performance.now() - t0 };
|
||||
}
|
||||
|
||||
async function measureRemember(
|
||||
baseUrl: string,
|
||||
rng: () => number,
|
||||
N: number,
|
||||
C: number,
|
||||
ops: number,
|
||||
): Promise<CellResult> {
|
||||
const { latencies, errors, wallMs } = await driveLoad(C, ops, async (i) => {
|
||||
const body = JSON.stringify({
|
||||
content: buildContent(rng, N + i),
|
||||
type: "observation",
|
||||
});
|
||||
const res = await fetch(`${baseUrl}/agentmemory/remember`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
await res.text().catch(() => "");
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
});
|
||||
return summarize("POST /agentmemory/remember", N, C, latencies, errors, wallMs);
|
||||
}
|
||||
|
||||
async function measureSmartSearch(
|
||||
baseUrl: string,
|
||||
rng: () => number,
|
||||
N: number,
|
||||
C: number,
|
||||
ops: number,
|
||||
): Promise<CellResult> {
|
||||
const queries = Array.from({ length: 32 }, (_, i) => buildContent(rng, i));
|
||||
const { latencies, errors, wallMs } = await driveLoad(C, ops, async (i) => {
|
||||
const body = JSON.stringify({
|
||||
query: queries[i % queries.length],
|
||||
limit: 5,
|
||||
});
|
||||
const res = await fetch(`${baseUrl}/agentmemory/smart-search`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
await res.text().catch(() => "");
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
});
|
||||
return summarize(
|
||||
"POST /agentmemory/smart-search",
|
||||
N,
|
||||
C,
|
||||
latencies,
|
||||
errors,
|
||||
wallMs,
|
||||
);
|
||||
}
|
||||
|
||||
async function measureMemoriesLatest(
|
||||
baseUrl: string,
|
||||
N: number,
|
||||
C: number,
|
||||
ops: number,
|
||||
): Promise<CellResult> {
|
||||
const { latencies, errors, wallMs } = await driveLoad(C, ops, async () => {
|
||||
const res = await fetch(`${baseUrl}/agentmemory/memories?latest=true`, {
|
||||
method: "GET",
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
await res.text().catch(() => "");
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
});
|
||||
return summarize(
|
||||
"GET /agentmemory/memories?latest=true",
|
||||
N,
|
||||
C,
|
||||
latencies,
|
||||
errors,
|
||||
wallMs,
|
||||
);
|
||||
}
|
||||
|
||||
interface RunReport {
|
||||
schema_version: 1;
|
||||
generated_at: string;
|
||||
git_sha: string;
|
||||
base_url: string;
|
||||
seed: number;
|
||||
matrix: { N: number[]; C: number[] };
|
||||
ops_per_cell: number;
|
||||
cells: CellResult[];
|
||||
notes: string;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const cfg = loadConfig();
|
||||
console.log(
|
||||
`[load-100k] base=${cfg.baseUrl} N=${cfg.Ns.join(",")} C=${cfg.Cs.join(",")} ops/cell=${cfg.opsPerCell} seed=${cfg.seed}`,
|
||||
);
|
||||
|
||||
let daemon: ChildProcess | null = null;
|
||||
if (cfg.autoStart) {
|
||||
console.log("[load-100k] AGENTMEMORY_BENCH_AUTOSTART=1 — spawning daemon");
|
||||
daemon = maybeStartDaemon();
|
||||
}
|
||||
|
||||
try {
|
||||
console.log("[load-100k] waiting for /agentmemory/livez (timeout 30s)");
|
||||
await waitForLivez(cfg.baseUrl, 30_000);
|
||||
|
||||
const cells: CellResult[] = [];
|
||||
// N sorted ascending so each cell builds on the previous seed work.
|
||||
const sortedNs = cfg.Ns.slice().sort((a, b) => a - b);
|
||||
let seededSoFar = 0;
|
||||
for (const N of sortedNs) {
|
||||
const delta = N - seededSoFar;
|
||||
if (delta > 0) {
|
||||
console.log(
|
||||
`[load-100k] seeding ${delta} memories (target N=${N})`,
|
||||
);
|
||||
const rng = mulberry32(cfg.seed + seededSoFar);
|
||||
const seedRes = await seedMemories(cfg.baseUrl, delta, rng);
|
||||
seededSoFar += seedRes.seeded;
|
||||
console.log(
|
||||
`[load-100k] seeded=${seedRes.seeded} errors=${seedRes.errors} wall=${(
|
||||
seedRes.wallMs / 1000
|
||||
).toFixed(2)}s`,
|
||||
);
|
||||
if (seedRes.errors > 0 && seedRes.seeded === 0) {
|
||||
throw new Error(
|
||||
`seeding produced 0 successes and ${seedRes.errors} errors — daemon misconfigured`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const C of cfg.Cs) {
|
||||
const probeRng = mulberry32(cfg.seed ^ (N * 0x9e3779b1) ^ C);
|
||||
|
||||
console.log(`[load-100k] cell N=${N} C=${C} remember`);
|
||||
const remember = await measureRemember(
|
||||
cfg.baseUrl,
|
||||
probeRng,
|
||||
N,
|
||||
C,
|
||||
cfg.opsPerCell,
|
||||
);
|
||||
cells.push(remember);
|
||||
|
||||
console.log(`[load-100k] cell N=${N} C=${C} smart-search`);
|
||||
const search = await measureSmartSearch(
|
||||
cfg.baseUrl,
|
||||
mulberry32(cfg.seed ^ (N * 0x85ebca77) ^ C),
|
||||
N,
|
||||
C,
|
||||
cfg.opsPerCell,
|
||||
);
|
||||
cells.push(search);
|
||||
|
||||
console.log(`[load-100k] cell N=${N} C=${C} memories?latest=true`);
|
||||
const memories = await measureMemoriesLatest(
|
||||
cfg.baseUrl,
|
||||
N,
|
||||
C,
|
||||
cfg.opsPerCell,
|
||||
);
|
||||
cells.push(memories);
|
||||
}
|
||||
}
|
||||
|
||||
const report: RunReport = {
|
||||
schema_version: 1,
|
||||
generated_at: new Date().toISOString(),
|
||||
git_sha: shortGitSha(),
|
||||
base_url: cfg.baseUrl,
|
||||
seed: cfg.seed,
|
||||
matrix: { N: sortedNs, C: cfg.Cs.slice() },
|
||||
ops_per_cell: cfg.opsPerCell,
|
||||
cells,
|
||||
notes:
|
||||
"Single-process load harness. Latency in milliseconds. " +
|
||||
"Throughput is wall-clock ops/sec for the cell (concurrent in-flight = C).",
|
||||
};
|
||||
|
||||
mkdirSync(cfg.outDir, { recursive: true });
|
||||
const outPath = join(cfg.outDir, `load-100k-${report.git_sha}.json`);
|
||||
writeFileSync(outPath, JSON.stringify(report, null, 2) + "\n", "utf8");
|
||||
console.log(`[load-100k] wrote ${outPath} (${cells.length} cells)`);
|
||||
|
||||
// Compact table to stdout for the verification commit.
|
||||
console.log("");
|
||||
console.log(
|
||||
[
|
||||
"endpoint".padEnd(40),
|
||||
"N".padStart(8),
|
||||
"C".padStart(4),
|
||||
"ops".padStart(6),
|
||||
"err".padStart(4),
|
||||
"p50_ms".padStart(8),
|
||||
"p90_ms".padStart(8),
|
||||
"p99_ms".padStart(8),
|
||||
"tp/s".padStart(9),
|
||||
].join(" "),
|
||||
);
|
||||
for (const c of cells) {
|
||||
console.log(
|
||||
[
|
||||
c.endpoint.padEnd(40),
|
||||
String(c.N).padStart(8),
|
||||
String(c.C).padStart(4),
|
||||
String(c.ops).padStart(6),
|
||||
String(c.errors).padStart(4),
|
||||
c.p50_ms.toFixed(2).padStart(8),
|
||||
c.p90_ms.toFixed(2).padStart(8),
|
||||
c.p99_ms.toFixed(2).padStart(8),
|
||||
c.throughput_per_sec.toFixed(2).padStart(9),
|
||||
].join(" "),
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
if (daemon) {
|
||||
daemon.kill("SIGTERM");
|
||||
// give it 2s to exit cleanly before SIGKILL
|
||||
await new Promise<void>((resolveFn) => {
|
||||
const t = setTimeout(() => {
|
||||
try {
|
||||
daemon!.kill("SIGKILL");
|
||||
} catch {
|
||||
/* already dead */
|
||||
}
|
||||
resolveFn();
|
||||
}, 2000);
|
||||
daemon!.once("exit", () => {
|
||||
clearTimeout(t);
|
||||
resolveFn();
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error("[load-100k] failed:", err instanceof Error ? err.stack : err);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,317 @@
|
||||
import { SearchIndex } from "../src/state/search-index.js";
|
||||
import { VectorIndex } from "../src/state/vector-index.js";
|
||||
import { HybridSearch } from "../src/state/hybrid-search.js";
|
||||
import type {
|
||||
CompressedObservation,
|
||||
EmbeddingProvider,
|
||||
} from "../src/types.js";
|
||||
import { readFileSync, writeFileSync, existsSync } from "node:fs";
|
||||
|
||||
interface LongMemEvalEntry {
|
||||
question_id: string;
|
||||
question_type: string;
|
||||
question: string;
|
||||
question_date: string;
|
||||
answer: string;
|
||||
answer_session_ids: string[];
|
||||
haystack_dates: string[];
|
||||
haystack_session_ids: string[];
|
||||
haystack_sessions: Array<Array<{ role: string; content: string; has_answer?: boolean }>>;
|
||||
}
|
||||
|
||||
interface SessionChunk {
|
||||
sessionId: string;
|
||||
text: string;
|
||||
turnCount: number;
|
||||
}
|
||||
|
||||
interface BenchResult {
|
||||
question_id: string;
|
||||
question_type: string;
|
||||
recall_any_at_5: number;
|
||||
recall_any_at_10: number;
|
||||
recall_any_at_20: number;
|
||||
ndcg_at_10: number;
|
||||
mrr: number;
|
||||
retrieved_session_ids: string[];
|
||||
gold_session_ids: string[];
|
||||
}
|
||||
|
||||
function chunkSessionToText(
|
||||
turns: Array<{ role: string; content: string }>,
|
||||
): string {
|
||||
return turns
|
||||
.map((t) => `${t.role}: ${t.content}`)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function recallAny(
|
||||
retrievedSessionIds: string[],
|
||||
goldSessionIds: string[],
|
||||
k: number,
|
||||
): number {
|
||||
const topK = new Set(retrievedSessionIds.slice(0, k));
|
||||
return goldSessionIds.some((gid) => topK.has(gid)) ? 1.0 : 0.0;
|
||||
}
|
||||
|
||||
function dcg(relevances: boolean[], k: number): number {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < Math.min(k, relevances.length); i++) {
|
||||
sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
|
||||
}
|
||||
return sum;
|
||||
}
|
||||
|
||||
function ndcg(
|
||||
retrievedSessionIds: string[],
|
||||
goldSessionIds: Set<string>,
|
||||
k: number,
|
||||
): number {
|
||||
const rels = retrievedSessionIds
|
||||
.slice(0, k)
|
||||
.map((id) => goldSessionIds.has(id));
|
||||
const idealRels = Array.from(
|
||||
{ length: Math.min(k, goldSessionIds.size) },
|
||||
() => true,
|
||||
);
|
||||
const idealDCG = dcg(idealRels, k);
|
||||
if (idealDCG === 0) return 0;
|
||||
return dcg(rels, k) / idealDCG;
|
||||
}
|
||||
|
||||
function mrr(
|
||||
retrievedSessionIds: string[],
|
||||
goldSessionIds: Set<string>,
|
||||
): number {
|
||||
for (let i = 0; i < retrievedSessionIds.length; i++) {
|
||||
if (goldSessionIds.has(retrievedSessionIds[i])) return 1 / (i + 1);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
class MockKV {
|
||||
private store = new Map<string, Map<string, unknown>>();
|
||||
async get<T>(scope: string, key: string): Promise<T> {
|
||||
const m = this.store.get(scope);
|
||||
if (!m || !m.has(key)) throw new Error(`Not found: ${scope}/${key}`);
|
||||
return m.get(key) as T;
|
||||
}
|
||||
async set(scope: string, key: string, value: unknown): Promise<void> {
|
||||
if (!this.store.has(scope)) this.store.set(scope, new Map());
|
||||
this.store.get(scope)!.set(key, value);
|
||||
}
|
||||
async list<T>(scope: string): Promise<T[]> {
|
||||
const m = this.store.get(scope);
|
||||
if (!m) return [];
|
||||
return Array.from(m.values()) as T[];
|
||||
}
|
||||
async delete(scope: string, key: string): Promise<void> {
|
||||
this.store.get(scope)?.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
async function runBenchmark(
|
||||
mode: "bm25" | "vector" | "hybrid",
|
||||
embeddingProvider?: EmbeddingProvider,
|
||||
) {
|
||||
const dataPath = new URL("./data/longmemeval_s_cleaned.json", import.meta.url).pathname;
|
||||
if (!existsSync(dataPath)) {
|
||||
console.error(`Dataset not found at ${dataPath}`);
|
||||
console.error("Download from: https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`Loading LongMemEval-S dataset...`);
|
||||
const raw = JSON.parse(readFileSync(dataPath, "utf-8")) as LongMemEvalEntry[];
|
||||
|
||||
const abstentionTypes = new Set([
|
||||
"single-session-user_abs",
|
||||
"multi-session_abs",
|
||||
"knowledge-update_abs",
|
||||
"temporal-reasoning_abs",
|
||||
]);
|
||||
const entries = raw.filter((e) => !abstentionTypes.has(e.question_type));
|
||||
console.log(
|
||||
`Loaded ${entries.length} questions (${raw.length - entries.length} abstention excluded)`,
|
||||
);
|
||||
|
||||
const results: BenchResult[] = [];
|
||||
let processed = 0;
|
||||
|
||||
for (const entry of entries) {
|
||||
const sessionChunks: SessionChunk[] = [];
|
||||
for (let i = 0; i < entry.haystack_sessions.length; i++) {
|
||||
const sessionId = entry.haystack_session_ids[i];
|
||||
const turns = entry.haystack_sessions[i];
|
||||
const text = chunkSessionToText(turns);
|
||||
sessionChunks.push({ sessionId, text, turnCount: turns.length });
|
||||
}
|
||||
|
||||
const bm25 = new SearchIndex();
|
||||
const vector = mode !== "bm25" ? new VectorIndex() : null;
|
||||
const kv = new MockKV();
|
||||
|
||||
const observations: CompressedObservation[] = [];
|
||||
for (const chunk of sessionChunks) {
|
||||
const obs: CompressedObservation = {
|
||||
id: `obs_${chunk.sessionId}`,
|
||||
sessionId: chunk.sessionId,
|
||||
timestamp: new Date().toISOString(),
|
||||
type: "conversation",
|
||||
title: chunk.text.slice(0, 80),
|
||||
facts: [],
|
||||
narrative: chunk.text,
|
||||
concepts: [],
|
||||
files: [],
|
||||
importance: 5,
|
||||
};
|
||||
observations.push(obs);
|
||||
bm25.add(obs);
|
||||
|
||||
if (vector && embeddingProvider) {
|
||||
try {
|
||||
const embedding = await embeddingProvider.embed(
|
||||
chunk.text.slice(0, 512),
|
||||
);
|
||||
vector.add(obs.id, chunk.sessionId, embedding);
|
||||
} catch {}
|
||||
}
|
||||
|
||||
await kv.set(`mem:obs:${chunk.sessionId}`, obs.id, obs);
|
||||
}
|
||||
|
||||
let retrievedObsIds: string[];
|
||||
|
||||
if (mode === "bm25") {
|
||||
const bm25Results = bm25.search(entry.question, 20);
|
||||
retrievedObsIds = bm25Results.map((r) => r.obsId);
|
||||
} else {
|
||||
const hybridSearch = new HybridSearch(
|
||||
bm25,
|
||||
vector,
|
||||
embeddingProvider || null,
|
||||
kv as any,
|
||||
0.4,
|
||||
0.6,
|
||||
0.0,
|
||||
false,
|
||||
);
|
||||
const hybridResults = await hybridSearch.search(entry.question, 20);
|
||||
retrievedObsIds = hybridResults.map((r) => r.observation.id);
|
||||
}
|
||||
|
||||
const retrievedSessionIds = retrievedObsIds.map((oid) =>
|
||||
oid.replace(/^obs_/, ""),
|
||||
);
|
||||
const goldSet = new Set(entry.answer_session_ids);
|
||||
|
||||
const result: BenchResult = {
|
||||
question_id: entry.question_id,
|
||||
question_type: entry.question_type,
|
||||
recall_any_at_5: recallAny(retrievedSessionIds, entry.answer_session_ids, 5),
|
||||
recall_any_at_10: recallAny(retrievedSessionIds, entry.answer_session_ids, 10),
|
||||
recall_any_at_20: recallAny(retrievedSessionIds, entry.answer_session_ids, 20),
|
||||
ndcg_at_10: ndcg(retrievedSessionIds, goldSet, 10),
|
||||
mrr: mrr(retrievedSessionIds, goldSet),
|
||||
retrieved_session_ids: retrievedSessionIds.slice(0, 10),
|
||||
gold_session_ids: entry.answer_session_ids,
|
||||
};
|
||||
results.push(result);
|
||||
processed++;
|
||||
|
||||
if (processed % 50 === 0) {
|
||||
const avgRecall5 =
|
||||
results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
|
||||
console.log(
|
||||
` [${processed}/${entries.length}] running recall_any@5: ${(avgRecall5 * 100).toFixed(1)}%`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const avgRecallAny5 =
|
||||
results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
|
||||
const avgRecallAny10 =
|
||||
results.reduce((s, r) => s + r.recall_any_at_10, 0) / results.length;
|
||||
const avgRecallAny20 =
|
||||
results.reduce((s, r) => s + r.recall_any_at_20, 0) / results.length;
|
||||
const avgNdcg10 =
|
||||
results.reduce((s, r) => s + r.ndcg_at_10, 0) / results.length;
|
||||
const avgMrr =
|
||||
results.reduce((s, r) => s + r.mrr, 0) / results.length;
|
||||
|
||||
const byType = new Map<string, BenchResult[]>();
|
||||
for (const r of results) {
|
||||
if (!byType.has(r.question_type)) byType.set(r.question_type, []);
|
||||
byType.get(r.question_type)!.push(r);
|
||||
}
|
||||
|
||||
console.log(`\n=== LongMemEval-S Results (${mode}) ===`);
|
||||
console.log(`Questions: ${results.length} (excl. abstention)`);
|
||||
console.log(`recall_any@5: ${(avgRecallAny5 * 100).toFixed(1)}%`);
|
||||
console.log(`recall_any@10: ${(avgRecallAny10 * 100).toFixed(1)}%`);
|
||||
console.log(`recall_any@20: ${(avgRecallAny20 * 100).toFixed(1)}%`);
|
||||
console.log(`NDCG@10: ${(avgNdcg10 * 100).toFixed(1)}%`);
|
||||
console.log(`MRR: ${(avgMrr * 100).toFixed(1)}%`);
|
||||
|
||||
console.log(`\nBy question type:`);
|
||||
for (const [type, typeResults] of byType) {
|
||||
const r5 =
|
||||
typeResults.reduce((s, r) => s + r.recall_any_at_5, 0) /
|
||||
typeResults.length;
|
||||
const r10 =
|
||||
typeResults.reduce((s, r) => s + r.recall_any_at_10, 0) /
|
||||
typeResults.length;
|
||||
console.log(
|
||||
` ${type.padEnd(30)} R@5: ${(r5 * 100).toFixed(1)}% R@10: ${(r10 * 100).toFixed(1)}% (n=${typeResults.length})`,
|
||||
);
|
||||
}
|
||||
|
||||
const outPath = new URL(
|
||||
`./data/longmemeval_results_${mode}.json`,
|
||||
import.meta.url,
|
||||
).pathname;
|
||||
writeFileSync(
|
||||
outPath,
|
||||
JSON.stringify(
|
||||
{
|
||||
mode,
|
||||
questions: results.length,
|
||||
recall_any_at_5: avgRecallAny5,
|
||||
recall_any_at_10: avgRecallAny10,
|
||||
recall_any_at_20: avgRecallAny20,
|
||||
ndcg_at_10: avgNdcg10,
|
||||
mrr: avgMrr,
|
||||
per_type: Object.fromEntries(
|
||||
Array.from(byType).map(([type, tr]) => [
|
||||
type,
|
||||
{
|
||||
count: tr.length,
|
||||
recall_any_at_5:
|
||||
tr.reduce((s, r) => s + r.recall_any_at_5, 0) / tr.length,
|
||||
recall_any_at_10:
|
||||
tr.reduce((s, r) => s + r.recall_any_at_10, 0) / tr.length,
|
||||
},
|
||||
]),
|
||||
),
|
||||
per_question: results,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
console.log(`\nResults saved to ${outPath}`);
|
||||
}
|
||||
|
||||
const mode = (process.argv[2] || "bm25") as "bm25" | "vector" | "hybrid";
|
||||
console.log(`Running LongMemEval-S benchmark in ${mode} mode...`);
|
||||
|
||||
if (mode === "bm25") {
|
||||
runBenchmark("bm25").catch(console.error);
|
||||
} else {
|
||||
import("../src/providers/embedding/local.js")
|
||||
.then(({ LocalEmbeddingProvider }) => {
|
||||
const provider = new LocalEmbeddingProvider();
|
||||
return runBenchmark(mode, provider);
|
||||
})
|
||||
.catch(console.error);
|
||||
}
|
||||
@@ -0,0 +1,643 @@
|
||||
import { SearchIndex } from "../src/state/search-index.js";
|
||||
import { VectorIndex } from "../src/state/vector-index.js";
|
||||
import { HybridSearch } from "../src/state/hybrid-search.js";
|
||||
import { GraphRetrieval } from "../src/functions/graph-retrieval.js";
|
||||
import { extractEntitiesFromQuery } from "../src/functions/query-expansion.js";
|
||||
import type { CompressedObservation, GraphNode, GraphEdge, GraphEdgeType } from "../src/types.js";
|
||||
import { generateDataset, type LabeledQuery } from "./dataset.js";
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
interface QualityMetrics {
|
||||
query: string;
|
||||
category: string;
|
||||
recall_at_5: number;
|
||||
recall_at_10: number;
|
||||
recall_at_20: number;
|
||||
precision_at_5: number;
|
||||
precision_at_10: number;
|
||||
ndcg_at_10: number;
|
||||
mrr: number;
|
||||
relevant_count: number;
|
||||
retrieved_count: number;
|
||||
latency_ms: number;
|
||||
}
|
||||
|
||||
interface SystemMetrics {
|
||||
system: string;
|
||||
avg_recall_at_5: number;
|
||||
avg_recall_at_10: number;
|
||||
avg_recall_at_20: number;
|
||||
avg_precision_at_5: number;
|
||||
avg_precision_at_10: number;
|
||||
avg_ndcg_at_10: number;
|
||||
avg_mrr: number;
|
||||
avg_latency_ms: number;
|
||||
total_tokens_per_query: number;
|
||||
per_query: QualityMetrics[];
|
||||
}
|
||||
|
||||
function dcg(relevances: boolean[], k: number): number {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < Math.min(k, relevances.length); i++) {
|
||||
sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
|
||||
}
|
||||
return sum;
|
||||
}
|
||||
|
||||
function ndcg(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
const actualRelevances = retrieved.slice(0, k).map(id => relevant.has(id));
|
||||
const idealRelevances = Array.from({ length: Math.min(k, relevant.size) }, () => true);
|
||||
const idealDCG = dcg(idealRelevances, k);
|
||||
if (idealDCG === 0) return 0;
|
||||
return dcg(actualRelevances, k) / idealDCG;
|
||||
}
|
||||
|
||||
function recall(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
if (relevant.size === 0) return 1;
|
||||
const topK = new Set(retrieved.slice(0, k));
|
||||
let hits = 0;
|
||||
for (const id of relevant) {
|
||||
if (topK.has(id)) hits++;
|
||||
}
|
||||
return hits / relevant.size;
|
||||
}
|
||||
|
||||
function precision(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
const topK = retrieved.slice(0, k);
|
||||
if (topK.length === 0) return 0;
|
||||
let hits = 0;
|
||||
for (const id of topK) {
|
||||
if (relevant.has(id)) hits++;
|
||||
}
|
||||
return hits / topK.length;
|
||||
}
|
||||
|
||||
function mrr(retrieved: string[], relevant: Set<string>): number {
|
||||
for (let i = 0; i < retrieved.length; i++) {
|
||||
if (relevant.has(retrieved[i])) return 1 / (i + 1);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function estimateTokens(text: string): number {
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
function mockKV() {
|
||||
const store = new Map<string, Map<string, unknown>>();
|
||||
return {
|
||||
get: async <T>(scope: string, key: string): Promise<T | null> => {
|
||||
return (store.get(scope)?.get(key) as T) ?? null;
|
||||
},
|
||||
set: async <T>(scope: string, key: string, data: T): Promise<T> => {
|
||||
if (!store.has(scope)) store.set(scope, new Map());
|
||||
store.get(scope)!.set(key, data);
|
||||
return data;
|
||||
},
|
||||
delete: async (scope: string, key: string): Promise<void> => {
|
||||
store.get(scope)?.delete(key);
|
||||
},
|
||||
list: async <T>(scope: string): Promise<T[]> => {
|
||||
const entries = store.get(scope);
|
||||
return entries ? (Array.from(entries.values()) as T[]) : [];
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function deterministicEmbedding(text: string, dims = 384): Float32Array {
|
||||
const arr = new Float32Array(dims);
|
||||
const words = text.toLowerCase().split(/\W+/).filter(w => w.length > 2);
|
||||
for (const word of words) {
|
||||
for (let i = 0; i < word.length; i++) {
|
||||
const idx = (word.charCodeAt(i) * 31 + i * 17) % dims;
|
||||
arr[idx] += 1;
|
||||
const idx2 = (word.charCodeAt(i) * 37 + i * 13 + word.length * 7) % dims;
|
||||
arr[idx2] += 0.5;
|
||||
}
|
||||
}
|
||||
const norm = Math.sqrt(arr.reduce((s, v) => s + v * v, 0));
|
||||
if (norm > 0) for (let i = 0; i < dims; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
}
|
||||
|
||||
async function evalBm25Only(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemMetrics> {
|
||||
const index = new SearchIndex();
|
||||
for (const obs of observations) index.add(obs);
|
||||
|
||||
const perQuery: QualityMetrics[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
const results = index.search(q.query, 20);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = results.map(r => r.obsId);
|
||||
perQuery.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_at_5: recall(retrieved, relevant, 5),
|
||||
recall_at_10: recall(retrieved, relevant, 10),
|
||||
recall_at_20: recall(retrieved, relevant, 20),
|
||||
precision_at_5: precision(retrieved, relevant, 5),
|
||||
precision_at_10: precision(retrieved, relevant, 10),
|
||||
ndcg_at_10: ndcg(retrieved, relevant, 10),
|
||||
mrr: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
retrieved_count: results.length,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
const avgTokens = perQuery.reduce((sum, q) => sum + q.retrieved_count, 0) / perQuery.length;
|
||||
const avgObsTokens = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
|
||||
|
||||
return {
|
||||
system: "BM25-only",
|
||||
avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
|
||||
avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
|
||||
avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
|
||||
avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
|
||||
avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
|
||||
avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
|
||||
avg_mrr: avg(perQuery.map(q => q.mrr)),
|
||||
avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
|
||||
total_tokens_per_query: Math.round(avgObsTokens * avgTokens),
|
||||
per_query: perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
async function evalDualStream(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemMetrics> {
|
||||
const kv = mockKV();
|
||||
const bm25 = new SearchIndex();
|
||||
const vector = new VectorIndex();
|
||||
const dims = 384;
|
||||
|
||||
for (const obs of observations) {
|
||||
bm25.add(obs);
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ");
|
||||
vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
|
||||
await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
|
||||
}
|
||||
|
||||
const mockEmbed: any = {
|
||||
name: "deterministic",
|
||||
dimensions: dims,
|
||||
embed: async (text: string) => deterministicEmbedding(text, dims),
|
||||
embedBatch: async (texts: string[]) => texts.map(t => deterministicEmbedding(t, dims)),
|
||||
};
|
||||
|
||||
const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
|
||||
const perQuery: QualityMetrics[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
const results = await hybrid.search(q.query, 20);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = results.map(r => r.observation.id);
|
||||
perQuery.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_at_5: recall(retrieved, relevant, 5),
|
||||
recall_at_10: recall(retrieved, relevant, 10),
|
||||
recall_at_20: recall(retrieved, relevant, 20),
|
||||
precision_at_5: precision(retrieved, relevant, 5),
|
||||
precision_at_10: precision(retrieved, relevant, 10),
|
||||
ndcg_at_10: ndcg(retrieved, relevant, 10),
|
||||
mrr: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
retrieved_count: results.length,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
const avgResultTokens = perQuery.reduce((sum, q) => {
|
||||
return sum + q.retrieved_count;
|
||||
}, 0) / perQuery.length;
|
||||
const avgObsTokens2 = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
|
||||
|
||||
return {
|
||||
system: "Dual-stream (BM25+Vector)",
|
||||
avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
|
||||
avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
|
||||
avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
|
||||
avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
|
||||
avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
|
||||
avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
|
||||
avg_mrr: avg(perQuery.map(q => q.mrr)),
|
||||
avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
|
||||
total_tokens_per_query: Math.round(avgObsTokens2 * avgResultTokens),
|
||||
per_query: perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
async function evalTripleStream(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemMetrics> {
|
||||
const kv = mockKV();
|
||||
const bm25 = new SearchIndex();
|
||||
const vector = new VectorIndex();
|
||||
const dims = 384;
|
||||
|
||||
for (const obs of observations) {
|
||||
bm25.add(obs);
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ");
|
||||
vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
|
||||
await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
|
||||
}
|
||||
|
||||
const conceptToNodes = new Map<string, string>();
|
||||
const nodeTypes: GraphNode["type"][] = ["concept", "library", "file", "pattern"];
|
||||
const edgeTypes: GraphEdgeType[] = ["uses", "related_to", "depends_on", "modifies"];
|
||||
const now = new Date().toISOString();
|
||||
let nodeId = 0;
|
||||
|
||||
for (const obs of observations) {
|
||||
for (const concept of obs.concepts) {
|
||||
if (!conceptToNodes.has(concept)) {
|
||||
const nid = `gn_${nodeId++}`;
|
||||
conceptToNodes.set(concept, nid);
|
||||
await kv.set("mem:graph:nodes", nid, {
|
||||
id: nid,
|
||||
type: nodeTypes[nodeId % nodeTypes.length],
|
||||
name: concept,
|
||||
properties: {},
|
||||
sourceObservationIds: [],
|
||||
createdAt: now,
|
||||
} as GraphNode);
|
||||
}
|
||||
const nid = conceptToNodes.get(concept)!;
|
||||
const existing = await kv.get<GraphNode>("mem:graph:nodes", nid);
|
||||
if (existing && !existing.sourceObservationIds.includes(obs.id)) {
|
||||
existing.sourceObservationIds.push(obs.id);
|
||||
await kv.set("mem:graph:nodes", nid, existing);
|
||||
}
|
||||
}
|
||||
|
||||
const capped = obs.concepts.slice(0, 10);
|
||||
for (let i = 0; i < capped.length; i++) {
|
||||
for (let j = i + 1; j < capped.length; j++) {
|
||||
const srcNid = conceptToNodes.get(capped[i])!;
|
||||
const tgtNid = conceptToNodes.get(capped[j])!;
|
||||
if (srcNid && tgtNid && srcNid !== tgtNid) {
|
||||
const eid = `ge_${srcNid}_${tgtNid}`;
|
||||
const existing = await kv.get<GraphEdge>("mem:graph:edges", eid);
|
||||
const weight = existing ? Math.min(1.0, existing.weight + 0.1) : 0.5;
|
||||
await kv.set("mem:graph:edges", eid, {
|
||||
id: eid,
|
||||
type: edgeTypes[(i + j) % edgeTypes.length],
|
||||
sourceNodeId: srcNid,
|
||||
targetNodeId: tgtNid,
|
||||
weight,
|
||||
sourceObservationIds: existing
|
||||
? [...new Set([...existing.sourceObservationIds, obs.id])]
|
||||
: [obs.id],
|
||||
createdAt: now,
|
||||
tcommit: now,
|
||||
version: 1,
|
||||
isLatest: true,
|
||||
} as GraphEdge);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const mockEmbed: any = {
|
||||
name: "deterministic",
|
||||
dimensions: dims,
|
||||
embed: async (text: string) => deterministicEmbedding(text, dims),
|
||||
embedBatch: async (texts: string[]) => texts.map(t => deterministicEmbedding(t, dims)),
|
||||
};
|
||||
|
||||
const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0.3);
|
||||
const perQuery: QualityMetrics[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
const results = await hybrid.search(q.query, 20);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = results.map(r => r.observation.id);
|
||||
perQuery.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_at_5: recall(retrieved, relevant, 5),
|
||||
recall_at_10: recall(retrieved, relevant, 10),
|
||||
recall_at_20: recall(retrieved, relevant, 20),
|
||||
precision_at_5: precision(retrieved, relevant, 5),
|
||||
precision_at_10: precision(retrieved, relevant, 10),
|
||||
ndcg_at_10: ndcg(retrieved, relevant, 10),
|
||||
mrr: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
retrieved_count: results.length,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
const avgResultTokens3 = perQuery.reduce((sum, q) => {
|
||||
return sum + q.retrieved_count;
|
||||
}, 0) / perQuery.length;
|
||||
const avgObsTokens3 = observations.slice(0, 50).reduce((s, o) => s + estimateTokens(JSON.stringify(o)), 0) / 50;
|
||||
|
||||
return {
|
||||
system: "Triple-stream (BM25+Vector+Graph)",
|
||||
avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
|
||||
avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
|
||||
avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
|
||||
avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
|
||||
avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
|
||||
avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
|
||||
avg_mrr: avg(perQuery.map(q => q.mrr)),
|
||||
avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
|
||||
total_tokens_per_query: Math.round(avgObsTokens3 * avgResultTokens3),
|
||||
per_query: perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
async function evalBuiltinMemory(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemMetrics> {
|
||||
const allText = observations.map(o =>
|
||||
`## ${o.title}\n${o.narrative}\nConcepts: ${o.concepts.join(", ")}\nFiles: ${o.files.join(", ")}`
|
||||
).join("\n\n");
|
||||
|
||||
const totalTokens = estimateTokens(allText);
|
||||
|
||||
const perQuery: QualityMetrics[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
|
||||
const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
|
||||
const scored: Array<{ id: string; score: number }> = [];
|
||||
|
||||
for (const obs of observations) {
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ").toLowerCase();
|
||||
let score = 0;
|
||||
for (const term of queryTerms) {
|
||||
if (text.includes(term)) score++;
|
||||
}
|
||||
if (score > 0) scored.push({ id: obs.id, score });
|
||||
}
|
||||
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = scored.map(s => s.id).slice(0, 20);
|
||||
perQuery.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_at_5: recall(retrieved, relevant, 5),
|
||||
recall_at_10: recall(retrieved, relevant, 10),
|
||||
recall_at_20: recall(retrieved, relevant, 20),
|
||||
precision_at_5: precision(retrieved, relevant, 5),
|
||||
precision_at_10: precision(retrieved, relevant, 10),
|
||||
ndcg_at_10: ndcg(retrieved, relevant, 10),
|
||||
mrr: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
retrieved_count: Math.min(scored.length, 20),
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
system: "Built-in (CLAUDE.md / grep)",
|
||||
avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
|
||||
avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
|
||||
avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
|
||||
avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
|
||||
avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
|
||||
avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
|
||||
avg_mrr: avg(perQuery.map(q => q.mrr)),
|
||||
avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
|
||||
total_tokens_per_query: totalTokens,
|
||||
per_query: perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
async function evalBuiltinMemoryTruncated(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemMetrics> {
|
||||
const MAX_LINES = 200;
|
||||
const lines = observations.map(o =>
|
||||
`- ${o.title}: ${o.narrative.slice(0, 80)}... [${o.concepts.slice(0, 3).join(", ")}]`
|
||||
);
|
||||
const truncated = lines.slice(0, MAX_LINES);
|
||||
const truncatedIds = new Set(observations.slice(0, MAX_LINES).map(o => o.id));
|
||||
const totalTokens = estimateTokens(truncated.join("\n"));
|
||||
|
||||
const perQuery: QualityMetrics[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
|
||||
const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
|
||||
const scored: Array<{ id: string; score: number }> = [];
|
||||
|
||||
for (let i = 0; i < Math.min(MAX_LINES, observations.length); i++) {
|
||||
const obs = observations[i];
|
||||
const line = truncated[i];
|
||||
let score = 0;
|
||||
for (const term of queryTerms) {
|
||||
if (line.toLowerCase().includes(term)) score++;
|
||||
}
|
||||
if (score > 0) scored.push({ id: obs.id, score });
|
||||
}
|
||||
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = scored.map(s => s.id).slice(0, 20);
|
||||
|
||||
const reachableRelevant = new Set(
|
||||
[...relevant].filter(id => truncatedIds.has(id))
|
||||
);
|
||||
|
||||
perQuery.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_at_5: recall(retrieved, relevant, 5),
|
||||
recall_at_10: recall(retrieved, relevant, 10),
|
||||
recall_at_20: recall(retrieved, relevant, 20),
|
||||
precision_at_5: precision(retrieved, relevant, 5),
|
||||
precision_at_10: precision(retrieved, relevant, 10),
|
||||
ndcg_at_10: ndcg(retrieved, relevant, 10),
|
||||
mrr: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
retrieved_count: Math.min(scored.length, 20),
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
system: "Built-in (200-line MEMORY.md)",
|
||||
avg_recall_at_5: avg(perQuery.map(q => q.recall_at_5)),
|
||||
avg_recall_at_10: avg(perQuery.map(q => q.recall_at_10)),
|
||||
avg_recall_at_20: avg(perQuery.map(q => q.recall_at_20)),
|
||||
avg_precision_at_5: avg(perQuery.map(q => q.precision_at_5)),
|
||||
avg_precision_at_10: avg(perQuery.map(q => q.precision_at_10)),
|
||||
avg_ndcg_at_10: avg(perQuery.map(q => q.ndcg_at_10)),
|
||||
avg_mrr: avg(perQuery.map(q => q.mrr)),
|
||||
avg_latency_ms: avg(perQuery.map(q => q.latency_ms)),
|
||||
total_tokens_per_query: totalTokens,
|
||||
per_query: perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
function avg(nums: number[]): number {
|
||||
return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;
|
||||
}
|
||||
|
||||
function pct(n: number): string {
|
||||
return (n * 100).toFixed(1) + "%";
|
||||
}
|
||||
|
||||
function generateReport(systems: SystemMetrics[], obsCount: number, queryCount: number): string {
|
||||
const lines: string[] = [];
|
||||
const w = (s: string) => lines.push(s);
|
||||
|
||||
w("# agentmemory v0.6.0 — Search Quality Evaluation");
|
||||
w("");
|
||||
w(`**Date:** ${new Date().toISOString()}`);
|
||||
w(`**Dataset:** ${obsCount} observations across 30 sessions (realistic coding project)`);
|
||||
w(`**Queries:** ${queryCount} labeled queries with ground-truth relevance`);
|
||||
w(`**Metric definitions:** Recall@K (fraction of relevant docs in top K), Precision@K (fraction of top K that are relevant), NDCG@10 (ranking quality), MRR (position of first relevant result)`);
|
||||
w("");
|
||||
|
||||
w("## Head-to-Head Comparison");
|
||||
w("");
|
||||
w("| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Latency | Tokens/query |");
|
||||
w("|--------|----------|-----------|-------------|---------|-----|---------|--------------|");
|
||||
for (const s of systems) {
|
||||
w(`| ${s.system} | ${pct(s.avg_recall_at_5)} | ${pct(s.avg_recall_at_10)} | ${pct(s.avg_precision_at_5)} | ${pct(s.avg_ndcg_at_10)} | ${pct(s.avg_mrr)} | ${s.avg_latency_ms.toFixed(2)}ms | ${s.total_tokens_per_query.toLocaleString()} |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Why This Matters");
|
||||
w("");
|
||||
|
||||
const builtin = systems.find(s => s.system.includes("CLAUDE.md / grep"));
|
||||
const truncated = systems.find(s => s.system.includes("200-line"));
|
||||
const triple = systems.find(s => s.system.includes("Triple"));
|
||||
const bm25 = systems.find(s => s.system === "BM25-only");
|
||||
|
||||
if (builtin && triple) {
|
||||
const recallLift = ((triple.avg_recall_at_10 - builtin.avg_recall_at_10) / Math.max(0.001, builtin.avg_recall_at_10) * 100);
|
||||
const tokenSaving = ((1 - triple.total_tokens_per_query / builtin.total_tokens_per_query) * 100);
|
||||
w(`**Recall improvement:** agentmemory triple-stream finds ${pct(triple.avg_recall_at_10)} of relevant memories at K=10 vs ${pct(builtin.avg_recall_at_10)} for keyword grep (${recallLift > 0 ? "+" : ""}${recallLift.toFixed(0)}%)`);
|
||||
w(`**Token savings:** agentmemory returns only the top 10 results (${triple.total_tokens_per_query.toLocaleString()} tokens) vs loading everything into context (${builtin.total_tokens_per_query.toLocaleString()} tokens) — ${tokenSaving.toFixed(0)}% reduction`);
|
||||
}
|
||||
|
||||
if (truncated && triple) {
|
||||
w(`**200-line cap:** Claude Code's MEMORY.md is capped at 200 lines. With ${obsCount} observations, ${pct(truncated.avg_recall_at_10)} recall at K=10 — memories from later sessions are simply invisible.`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Per-Query Breakdown (Triple-Stream)");
|
||||
w("");
|
||||
|
||||
if (triple) {
|
||||
w("| Query | Category | Recall@10 | NDCG@10 | MRR | Relevant | Latency |");
|
||||
w("|-------|----------|-----------|---------|-----|----------|---------|");
|
||||
for (const q of triple.per_query) {
|
||||
w(`| ${q.query.slice(0, 45)}${q.query.length > 45 ? "..." : ""} | ${q.category} | ${pct(q.recall_at_10)} | ${pct(q.ndcg_at_10)} | ${pct(q.mrr)} | ${q.relevant_count} | ${q.latency_ms.toFixed(1)}ms |`);
|
||||
}
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## By Query Category");
|
||||
w("");
|
||||
|
||||
const categories = ["exact", "semantic", "cross-session", "entity"];
|
||||
if (triple) {
|
||||
w("| Category | Avg Recall@10 | Avg NDCG@10 | Avg MRR | Queries |");
|
||||
w("|----------|---------------|-------------|---------|---------|");
|
||||
for (const cat of categories) {
|
||||
const qs = triple.per_query.filter(q => q.category === cat);
|
||||
if (qs.length === 0) continue;
|
||||
w(`| ${cat} | ${pct(avg(qs.map(q => q.recall_at_10)))} | ${pct(avg(qs.map(q => q.ndcg_at_10)))} | ${pct(avg(qs.map(q => q.mrr)))} | ${qs.length} |`);
|
||||
}
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Context Window Analysis");
|
||||
w("");
|
||||
w("The fundamental problem with built-in agent memory:");
|
||||
w("");
|
||||
w("| Observations | MEMORY.md tokens | agentmemory tokens (top 10) | Savings | MEMORY.md reachable |");
|
||||
w("|-------------|-----------------|---------------------------|---------|-------------------|");
|
||||
|
||||
for (const count of [240, 500, 1000, 5000]) {
|
||||
const memTokens = Math.round(count * 50);
|
||||
const amTokens = triple ? triple.total_tokens_per_query : 500;
|
||||
const saving = ((1 - amTokens / memTokens) * 100);
|
||||
const reachable = count <= 200 ? "100%" : `${((200 / count) * 100).toFixed(0)}%`;
|
||||
w(`| ${count.toLocaleString()} | ${memTokens.toLocaleString()} | ${amTokens.toLocaleString()} | ${saving.toFixed(0)}% | ${reachable} |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("At 240 observations (our dataset), MEMORY.md already hits its 200-line cap and loses access to the most recent 40 observations. At 1,000 observations, 80% of memories are invisible. agentmemory always searches the full corpus.");
|
||||
|
||||
w("");
|
||||
w("---");
|
||||
w("");
|
||||
w(`*${systems.reduce((s, sys) => s + sys.per_query.length, 0)} evaluations across ${systems.length} systems. Ground-truth labels assigned by concept matching against observation metadata.*`);
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log("Generating labeled dataset...");
|
||||
const { observations, queries, sessions } = generateDataset();
|
||||
console.log(`Dataset: ${observations.length} observations, ${sessions.size} sessions, ${queries.length} queries`);
|
||||
console.log(`Avg relevant docs per query: ${(queries.reduce((s, q) => s + q.relevantObsIds.length, 0) / queries.length).toFixed(1)}`);
|
||||
console.log("");
|
||||
|
||||
console.log("Evaluating: Built-in (CLAUDE.md / grep)...");
|
||||
const builtinResults = await evalBuiltinMemory(observations, queries);
|
||||
console.log(` Recall@10: ${pct(builtinResults.avg_recall_at_10)}, NDCG@10: ${pct(builtinResults.avg_ndcg_at_10)}`);
|
||||
|
||||
console.log("Evaluating: Built-in (200-line MEMORY.md)...");
|
||||
const truncatedResults = await evalBuiltinMemoryTruncated(observations, queries);
|
||||
console.log(` Recall@10: ${pct(truncatedResults.avg_recall_at_10)}, NDCG@10: ${pct(truncatedResults.avg_ndcg_at_10)}`);
|
||||
|
||||
console.log("Evaluating: BM25-only...");
|
||||
const bm25Results = await evalBm25Only(observations, queries);
|
||||
console.log(` Recall@10: ${pct(bm25Results.avg_recall_at_10)}, NDCG@10: ${pct(bm25Results.avg_ndcg_at_10)}`);
|
||||
|
||||
console.log("Evaluating: Dual-stream (BM25+Vector)...");
|
||||
const dualResults = await evalDualStream(observations, queries);
|
||||
console.log(` Recall@10: ${pct(dualResults.avg_recall_at_10)}, NDCG@10: ${pct(dualResults.avg_ndcg_at_10)}`);
|
||||
|
||||
console.log("Evaluating: Triple-stream (BM25+Vector+Graph)...");
|
||||
const tripleResults = await evalTripleStream(observations, queries);
|
||||
console.log(` Recall@10: ${pct(tripleResults.avg_recall_at_10)}, NDCG@10: ${pct(tripleResults.avg_ndcg_at_10)}`);
|
||||
|
||||
console.log("");
|
||||
|
||||
const report = generateReport(
|
||||
[builtinResults, truncatedResults, bm25Results, dualResults, tripleResults],
|
||||
observations.length,
|
||||
queries.length,
|
||||
);
|
||||
|
||||
writeFileSync("benchmark/QUALITY.md", report);
|
||||
console.log(report);
|
||||
console.log(`\nReport written to benchmark/QUALITY.md`);
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
@@ -0,0 +1,405 @@
|
||||
import { SearchIndex } from "../src/state/search-index.js";
|
||||
import { VectorIndex } from "../src/state/vector-index.js";
|
||||
import { HybridSearch } from "../src/state/hybrid-search.js";
|
||||
import { LocalEmbeddingProvider } from "../src/providers/embedding/local.js";
|
||||
import type { CompressedObservation, EmbeddingProvider } from "../src/types.js";
|
||||
import { generateDataset, type LabeledQuery } from "./dataset.js";
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
function mockKV() {
|
||||
const store = new Map<string, Map<string, unknown>>();
|
||||
return {
|
||||
get: async <T>(scope: string, key: string): Promise<T | null> =>
|
||||
(store.get(scope)?.get(key) as T) ?? null,
|
||||
set: async <T>(scope: string, key: string, data: T): Promise<T> => {
|
||||
if (!store.has(scope)) store.set(scope, new Map());
|
||||
store.get(scope)!.set(key, data);
|
||||
return data;
|
||||
},
|
||||
delete: async (scope: string, key: string): Promise<void> => {
|
||||
store.get(scope)?.delete(key);
|
||||
},
|
||||
list: async <T>(scope: string): Promise<T[]> => {
|
||||
const entries = store.get(scope);
|
||||
return entries ? (Array.from(entries.values()) as T[]) : [];
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function estimateTokens(text: string): number {
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
function obsToText(obs: CompressedObservation): string {
|
||||
return [obs.title, obs.subtitle || "", obs.narrative, ...obs.facts, ...obs.concepts].join(" ");
|
||||
}
|
||||
|
||||
function recall(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
if (relevant.size === 0) return 1;
|
||||
const topK = new Set(retrieved.slice(0, k));
|
||||
let hits = 0;
|
||||
for (const id of relevant) if (topK.has(id)) hits++;
|
||||
return hits / relevant.size;
|
||||
}
|
||||
|
||||
function precision(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
const topK = retrieved.slice(0, k);
|
||||
if (topK.length === 0) return 0;
|
||||
let hits = 0;
|
||||
for (const id of topK) if (relevant.has(id)) hits++;
|
||||
return hits / topK.length;
|
||||
}
|
||||
|
||||
function dcg(relevances: boolean[], k: number): number {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < Math.min(k, relevances.length); i++)
|
||||
sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
|
||||
return sum;
|
||||
}
|
||||
|
||||
function ndcg(retrieved: string[], relevant: Set<string>, k: number): number {
|
||||
const actual = retrieved.slice(0, k).map(id => relevant.has(id));
|
||||
const ideal = Array.from({ length: Math.min(k, relevant.size) }, () => true);
|
||||
const idealDCG = dcg(ideal, k);
|
||||
return idealDCG === 0 ? 0 : dcg(actual, k) / idealDCG;
|
||||
}
|
||||
|
||||
function mrr(retrieved: string[], relevant: Set<string>): number {
|
||||
for (let i = 0; i < retrieved.length; i++)
|
||||
if (relevant.has(retrieved[i])) return 1 / (i + 1);
|
||||
return 0;
|
||||
}
|
||||
|
||||
function avg(nums: number[]): number {
|
||||
return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;
|
||||
}
|
||||
|
||||
function pct(n: number): string {
|
||||
return (n * 100).toFixed(1) + "%";
|
||||
}
|
||||
|
||||
interface QueryResult {
|
||||
query: string;
|
||||
category: string;
|
||||
recall_5: number;
|
||||
recall_10: number;
|
||||
precision_5: number;
|
||||
ndcg_10: number;
|
||||
mrr_val: number;
|
||||
relevant_count: number;
|
||||
latency_ms: number;
|
||||
}
|
||||
|
||||
interface SystemResult {
|
||||
name: string;
|
||||
results: QueryResult[];
|
||||
embed_time_ms: number;
|
||||
tokens_per_query: number;
|
||||
}
|
||||
|
||||
async function evalSystem(
|
||||
name: string,
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
provider: EmbeddingProvider | null,
|
||||
weights: { bm25: number; vector: number; graph: number },
|
||||
): Promise<SystemResult> {
|
||||
const kv = mockKV();
|
||||
const bm25 = new SearchIndex();
|
||||
const vector = provider ? new VectorIndex() : null;
|
||||
|
||||
console.log(` Indexing ${observations.length} observations...`);
|
||||
const embedStart = performance.now();
|
||||
|
||||
for (const obs of observations) {
|
||||
bm25.add(obs);
|
||||
await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
|
||||
}
|
||||
|
||||
if (provider && vector) {
|
||||
const batchSize = 32;
|
||||
for (let i = 0; i < observations.length; i += batchSize) {
|
||||
const batch = observations.slice(i, i + batchSize);
|
||||
const texts = batch.map(o => obsToText(o));
|
||||
const embeddings = await provider.embedBatch(texts);
|
||||
for (let j = 0; j < batch.length; j++) {
|
||||
vector.add(batch[j].id, batch[j].sessionId, embeddings[j]);
|
||||
}
|
||||
if ((i + batchSize) % 100 === 0 || i + batchSize >= observations.length) {
|
||||
process.stdout.write(`\r Embedded ${Math.min(i + batchSize, observations.length)}/${observations.length}`);
|
||||
}
|
||||
}
|
||||
console.log("");
|
||||
}
|
||||
|
||||
const embedTime = performance.now() - embedStart;
|
||||
|
||||
const hybrid = new HybridSearch(
|
||||
bm25,
|
||||
vector,
|
||||
provider,
|
||||
kv as never,
|
||||
weights.bm25,
|
||||
weights.vector,
|
||||
weights.graph,
|
||||
);
|
||||
|
||||
console.log(` Running ${queries.length} queries...`);
|
||||
const results: QueryResult[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const start = performance.now();
|
||||
const searchResults = await hybrid.search(q.query, 20);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = searchResults.map(r => r.observation.id);
|
||||
results.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_5: recall(retrieved, relevant, 5),
|
||||
recall_10: recall(retrieved, relevant, 10),
|
||||
precision_5: precision(retrieved, relevant, 5),
|
||||
ndcg_10: ndcg(retrieved, relevant, 10),
|
||||
mrr_val: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
let totalReturnedTokens = 0;
|
||||
for (const q of queries) {
|
||||
const searchResults = await hybrid.search(q.query, 10);
|
||||
totalReturnedTokens += searchResults.reduce(
|
||||
(sum, r) => sum + estimateTokens(JSON.stringify(r.observation)),
|
||||
0,
|
||||
);
|
||||
}
|
||||
const avgReturnedTokens = Math.round(totalReturnedTokens / queries.length);
|
||||
|
||||
return {
|
||||
name,
|
||||
results,
|
||||
embed_time_ms: embedTime,
|
||||
tokens_per_query: avgReturnedTokens,
|
||||
};
|
||||
}
|
||||
|
||||
async function evalBuiltinGrep(
|
||||
observations: CompressedObservation[],
|
||||
queries: LabeledQuery[],
|
||||
): Promise<SystemResult> {
|
||||
const results: QueryResult[] = [];
|
||||
|
||||
for (const q of queries) {
|
||||
const relevant = new Set(q.relevantObsIds);
|
||||
const queryTerms = q.query.toLowerCase().split(/\W+/).filter(w => w.length > 2);
|
||||
const start = performance.now();
|
||||
|
||||
const scored: Array<{ id: string; score: number }> = [];
|
||||
for (const obs of observations) {
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts, ...obs.facts].join(" ").toLowerCase();
|
||||
let score = 0;
|
||||
for (const term of queryTerms) if (text.includes(term)) score++;
|
||||
if (score > 0) scored.push({ id: obs.id, score });
|
||||
}
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const retrieved = scored.map(s => s.id).slice(0, 20);
|
||||
results.push({
|
||||
query: q.query,
|
||||
category: q.category,
|
||||
recall_5: recall(retrieved, relevant, 5),
|
||||
recall_10: recall(retrieved, relevant, 10),
|
||||
precision_5: precision(retrieved, relevant, 5),
|
||||
ndcg_10: ndcg(retrieved, relevant, 10),
|
||||
mrr_val: mrr(retrieved, relevant),
|
||||
relevant_count: relevant.size,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
const allTokens = estimateTokens(observations.map(o =>
|
||||
`## ${o.title}\n${o.narrative}\nConcepts: ${o.concepts.join(", ")}`
|
||||
).join("\n\n"));
|
||||
|
||||
return { name: "Built-in (grep all)", results, embed_time_ms: 0, tokens_per_query: allTokens };
|
||||
}
|
||||
|
||||
function generateReport(systems: SystemResult[], obsCount: number): string {
|
||||
const lines: string[] = [];
|
||||
const w = (s: string) => lines.push(s);
|
||||
|
||||
w("# agentmemory v0.6.0 — Real Embeddings Quality Evaluation");
|
||||
w("");
|
||||
w(`**Date:** ${new Date().toISOString()}`);
|
||||
w(`**Platform:** ${process.platform} ${process.arch}, Node ${process.version}`);
|
||||
w(`**Dataset:** ${obsCount} observations, 30 sessions, 20 labeled queries`);
|
||||
w(`**Embedding model:** Xenova/all-MiniLM-L6-v2 (384d, local, no API key)`);
|
||||
w("");
|
||||
|
||||
w("## Head-to-Head: Real Embeddings vs Keyword Search");
|
||||
w("");
|
||||
w("| System | Recall@5 | Recall@10 | Precision@5 | NDCG@10 | MRR | Avg Latency | Tokens/query |");
|
||||
w("|--------|----------|-----------|-------------|---------|-----|-------------|--------------|");
|
||||
|
||||
for (const s of systems) {
|
||||
const r = s.results;
|
||||
w(`| ${s.name} | ${pct(avg(r.map(q => q.recall_5)))} | ${pct(avg(r.map(q => q.recall_10)))} | ${pct(avg(r.map(q => q.precision_5)))} | ${pct(avg(r.map(q => q.ndcg_10)))} | ${pct(avg(r.map(q => q.mrr_val)))} | ${avg(r.map(q => q.latency_ms)).toFixed(2)}ms | ${s.tokens_per_query.toLocaleString()} |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Improvement from Real Embeddings");
|
||||
w("");
|
||||
|
||||
const bm25Only = systems.find(s => s.name === "BM25-only (stemmed+synonyms)");
|
||||
const dual = systems.find(s => s.name.includes("Dual-stream"));
|
||||
const triple = systems.find(s => s.name.includes("Triple-stream"));
|
||||
const builtin = systems.find(s => s.name.includes("grep"));
|
||||
|
||||
if (bm25Only && dual) {
|
||||
const recallDelta = avg(dual.results.map(q => q.recall_10)) - avg(bm25Only.results.map(q => q.recall_10));
|
||||
w(`Adding real vector embeddings to BM25 improves recall@10 by **${(recallDelta * 100).toFixed(1)} percentage points**.`);
|
||||
}
|
||||
if (builtin && dual) {
|
||||
const tokenSaving = (1 - dual.tokens_per_query / builtin.tokens_per_query) * 100;
|
||||
w(`Token savings vs loading everything: **${tokenSaving.toFixed(0)}%** (${dual.tokens_per_query.toLocaleString()} vs ${builtin.tokens_per_query.toLocaleString()} tokens).`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Per-Query: Where Real Embeddings Win");
|
||||
w("");
|
||||
|
||||
if (bm25Only && dual) {
|
||||
w("Queries where dual-stream (real embeddings) outperforms BM25-only:");
|
||||
w("");
|
||||
w("| Query | Category | BM25 Recall@10 | +Vector Recall@10 | Delta |");
|
||||
w("|-------|----------|---------------|-------------------|-------|");
|
||||
|
||||
for (let i = 0; i < bm25Only.results.length; i++) {
|
||||
const bq = bm25Only.results[i];
|
||||
const dq = dual.results[i];
|
||||
const delta = dq.recall_10 - bq.recall_10;
|
||||
const marker = delta > 0 ? " **" : delta < 0 ? " *" : "";
|
||||
if (Math.abs(delta) > 0.001) {
|
||||
w(`| ${bq.query.slice(0, 45)}${bq.query.length > 45 ? "..." : ""} | ${bq.category} | ${pct(bq.recall_10)} | ${pct(dq.recall_10)} | ${delta > 0 ? "+" : ""}${(delta * 100).toFixed(1)}pp${marker} |`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## By Category Comparison");
|
||||
w("");
|
||||
const categories = ["exact", "semantic", "cross-session", "entity"];
|
||||
|
||||
w("| Category | Built-in grep | BM25 (stemmed) | +Real Vectors | +Graph |");
|
||||
w("|----------|--------------|----------------|--------------|--------|");
|
||||
|
||||
for (const cat of categories) {
|
||||
const vals = systems.map(s => {
|
||||
const qs = s.results.filter(q => q.category === cat);
|
||||
return qs.length ? pct(avg(qs.map(q => q.recall_10))) : "-";
|
||||
});
|
||||
w(`| ${cat} | ${vals.join(" | ")} |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Embedding Performance");
|
||||
w("");
|
||||
w("| System | Embedding Time | Model | Dimensions |");
|
||||
w("|--------|---------------|-------|------------|");
|
||||
for (const s of systems) {
|
||||
if (s.embed_time_ms > 100) {
|
||||
w(`| ${s.name} | ${(s.embed_time_ms / 1000).toFixed(1)}s | Xenova/all-MiniLM-L6-v2 | 384 |`);
|
||||
}
|
||||
}
|
||||
w("");
|
||||
w("Embedding is a one-time cost at ingestion. Search is sub-millisecond after indexing.");
|
||||
|
||||
w("");
|
||||
w("## Key Findings");
|
||||
w("");
|
||||
|
||||
if (bm25Only && dual) {
|
||||
const semBm25 = bm25Only.results.filter(q => q.category === "semantic");
|
||||
const semDual = dual.results.filter(q => q.category === "semantic");
|
||||
const semImprove = avg(semDual.map(q => q.recall_10)) - avg(semBm25.map(q => q.recall_10));
|
||||
|
||||
w(`1. **Semantic queries improve most**: ${(semImprove * 100).toFixed(1)}pp recall@10 gain from real embeddings`);
|
||||
w(`2. **"database performance optimization"** — the hardest query — goes from BM25 ${pct(bm25Only.results.find(q => q.query.includes("database perf"))?.recall_10 ?? 0)} to vector-augmented ${pct(dual.results.find(q => q.query.includes("database perf"))?.recall_10 ?? 0)}`);
|
||||
w(`3. **Entity/exact queries** are already well-served by BM25+stemming — vectors add marginal value`);
|
||||
w(`4. **Local embeddings (Xenova)** run without API keys — zero cost, zero latency concerns`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## Recommendation");
|
||||
w("");
|
||||
w("Enable local embeddings by default (`EMBEDDING_PROVIDER=local` or install `@xenova/transformers`).");
|
||||
w("This gives agentmemory genuine semantic search that built-in agent memories cannot match —");
|
||||
w("understanding that \"database performance optimization\" relates to \"N+1 query fix\" and \"eager loading\".");
|
||||
w("");
|
||||
|
||||
w("---");
|
||||
w(`*All measurements use Xenova/all-MiniLM-L6-v2 local embeddings (384 dimensions, no API calls).*`);
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log("=== agentmemory Real Embeddings Benchmark ===\n");
|
||||
|
||||
console.log("Loading Xenova/all-MiniLM-L6-v2 model (first run downloads ~80MB)...");
|
||||
let provider: EmbeddingProvider;
|
||||
try {
|
||||
provider = new LocalEmbeddingProvider();
|
||||
const testEmbed = await provider.embed("test");
|
||||
console.log(`Model loaded. Dimensions: ${testEmbed.length}\n`);
|
||||
} catch (err) {
|
||||
console.error("Failed to load Xenova model:", err);
|
||||
console.error("Install with: npm install @xenova/transformers");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { observations, queries } = generateDataset();
|
||||
console.log(`Dataset: ${observations.length} observations, ${queries.length} queries\n`);
|
||||
|
||||
console.log("1. Built-in (grep all)...");
|
||||
const builtinResult = await evalBuiltinGrep(observations, queries);
|
||||
console.log(` Recall@10: ${pct(avg(builtinResult.results.map(q => q.recall_10)))}\n`);
|
||||
|
||||
console.log("2. BM25-only (stemmed+synonyms)...");
|
||||
const bm25Result = await evalSystem(
|
||||
"BM25-only (stemmed+synonyms)",
|
||||
observations, queries, null,
|
||||
{ bm25: 1.0, vector: 0, graph: 0 },
|
||||
);
|
||||
console.log(` Recall@10: ${pct(avg(bm25Result.results.map(q => q.recall_10)))}\n`);
|
||||
|
||||
console.log("3. Dual-stream (BM25 + real Xenova vectors)...");
|
||||
const dualResult = await evalSystem(
|
||||
"Dual-stream (BM25+Xenova)",
|
||||
observations, queries, provider,
|
||||
{ bm25: 0.4, vector: 0.6, graph: 0 },
|
||||
);
|
||||
console.log(` Recall@10: ${pct(avg(dualResult.results.map(q => q.recall_10)))}\n`);
|
||||
|
||||
console.log("4. Triple-stream (BM25 + Xenova + Graph)...");
|
||||
const tripleResult = await evalSystem(
|
||||
"Triple-stream (BM25+Xenova+Graph)",
|
||||
observations, queries, provider,
|
||||
{ bm25: 0.4, vector: 0.6, graph: 0.3 },
|
||||
);
|
||||
console.log(` Recall@10: ${pct(avg(tripleResult.results.map(q => q.recall_10)))}\n`);
|
||||
|
||||
const report = generateReport(
|
||||
[builtinResult, bm25Result, dualResult, tripleResult],
|
||||
observations.length,
|
||||
);
|
||||
|
||||
writeFileSync("benchmark/REAL-EMBEDDINGS.md", report);
|
||||
console.log(report);
|
||||
console.log(`\nReport written to benchmark/REAL-EMBEDDINGS.md`);
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"generated_at": "2026-05-13T19:49:26.116Z",
|
||||
"git_sha": "96c0ed0",
|
||||
"base_url": "http://localhost:3111",
|
||||
"seed": 12648430,
|
||||
"matrix": {
|
||||
"N": [
|
||||
1000
|
||||
],
|
||||
"C": [
|
||||
10
|
||||
]
|
||||
},
|
||||
"ops_per_cell": 200,
|
||||
"cells": [
|
||||
{
|
||||
"endpoint": "POST /agentmemory/remember",
|
||||
"N": 1000,
|
||||
"C": 10,
|
||||
"ops": 200,
|
||||
"errors": 0,
|
||||
"wall_ms": 11504.64,
|
||||
"throughput_per_sec": 17.38,
|
||||
"p50_ms": 577.435,
|
||||
"p90_ms": 607.335,
|
||||
"p99_ms": 675.269,
|
||||
"min_ms": 64.46,
|
||||
"max_ms": 683.164
|
||||
},
|
||||
{
|
||||
"endpoint": "POST /agentmemory/smart-search",
|
||||
"N": 1000,
|
||||
"C": 10,
|
||||
"ops": 200,
|
||||
"errors": 0,
|
||||
"wall_ms": 3264.572,
|
||||
"throughput_per_sec": 61.26,
|
||||
"p50_ms": 160.064,
|
||||
"p90_ms": 185.608,
|
||||
"p99_ms": 224.354,
|
||||
"min_ms": 98.498,
|
||||
"max_ms": 251.317
|
||||
},
|
||||
{
|
||||
"endpoint": "GET /agentmemory/memories?latest=true",
|
||||
"N": 1000,
|
||||
"C": 10,
|
||||
"ops": 200,
|
||||
"errors": 0,
|
||||
"wall_ms": 8051.764,
|
||||
"throughput_per_sec": 24.84,
|
||||
"p50_ms": 395.462,
|
||||
"p90_ms": 475.714,
|
||||
"p99_ms": 542.648,
|
||||
"min_ms": 158.79,
|
||||
"max_ms": 635.331
|
||||
}
|
||||
],
|
||||
"notes": "Single-process load harness. Latency in milliseconds. Throughput is wall-clock ops/sec for the cell (concurrent in-flight = C)."
|
||||
}
|
||||
@@ -0,0 +1,398 @@
|
||||
import { SearchIndex } from "../src/state/search-index.js";
|
||||
import { VectorIndex } from "../src/state/vector-index.js";
|
||||
import { HybridSearch } from "../src/state/hybrid-search.js";
|
||||
import type { CompressedObservation } from "../src/types.js";
|
||||
import { generateScaleDataset, generateDataset } from "./dataset.js";
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
function mockKV() {
|
||||
const store = new Map<string, Map<string, unknown>>();
|
||||
return {
|
||||
get: async <T>(scope: string, key: string): Promise<T | null> =>
|
||||
(store.get(scope)?.get(key) as T) ?? null,
|
||||
set: async <T>(scope: string, key: string, data: T): Promise<T> => {
|
||||
if (!store.has(scope)) store.set(scope, new Map());
|
||||
store.get(scope)!.set(key, data);
|
||||
return data;
|
||||
},
|
||||
delete: async (scope: string, key: string): Promise<void> => {
|
||||
store.get(scope)?.delete(key);
|
||||
},
|
||||
list: async <T>(scope: string): Promise<T[]> => {
|
||||
const entries = store.get(scope);
|
||||
return entries ? (Array.from(entries.values()) as T[]) : [];
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function deterministicEmbedding(text: string, dims = 384): Float32Array {
|
||||
const arr = new Float32Array(dims);
|
||||
const words = text.toLowerCase().split(/\W+/).filter(w => w.length > 2);
|
||||
for (const word of words) {
|
||||
for (let i = 0; i < word.length; i++) {
|
||||
const idx = (word.charCodeAt(i) * 31 + i * 17) % dims;
|
||||
arr[idx] += 1;
|
||||
const idx2 = (word.charCodeAt(i) * 37 + i * 13 + word.length * 7) % dims;
|
||||
arr[idx2] += 0.5;
|
||||
}
|
||||
}
|
||||
const norm = Math.sqrt(arr.reduce((s, v) => s + v * v, 0));
|
||||
if (norm > 0) for (let i = 0; i < dims; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
}
|
||||
|
||||
function estimateTokens(text: string): number {
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
interface ScaleResult {
|
||||
scale: number;
|
||||
sessions: number;
|
||||
index_build_ms: number;
|
||||
index_build_per_doc_ms: number;
|
||||
bm25_search_ms: number;
|
||||
hybrid_search_ms: number;
|
||||
index_size_kb: number;
|
||||
vector_size_kb: number;
|
||||
heap_mb: number;
|
||||
builtin_tokens: number;
|
||||
builtin_200line_tokens: number;
|
||||
agentmemory_tokens: number;
|
||||
token_savings_pct: number;
|
||||
builtin_unreachable_pct: number;
|
||||
}
|
||||
|
||||
interface CrossSessionResult {
|
||||
query: string;
|
||||
target_session: string;
|
||||
current_session: string;
|
||||
sessions_apart: number;
|
||||
bm25_found: boolean;
|
||||
bm25_rank: number;
|
||||
hybrid_found: boolean;
|
||||
hybrid_rank: number;
|
||||
builtin_found: boolean;
|
||||
latency_ms: number;
|
||||
}
|
||||
|
||||
const SEARCH_QUERIES = [
|
||||
"authentication middleware JWT",
|
||||
"PostgreSQL connection pooling",
|
||||
"Kubernetes pod crash",
|
||||
"rate limiting API",
|
||||
"Playwright E2E tests",
|
||||
"Docker multi-stage build",
|
||||
"Redis caching layer",
|
||||
"CI/CD GitHub Actions",
|
||||
"Prisma migration drift",
|
||||
"monitoring Datadog alerts",
|
||||
];
|
||||
|
||||
async function benchmarkScale(counts: number[]): Promise<ScaleResult[]> {
|
||||
const results: ScaleResult[] = [];
|
||||
|
||||
for (const count of counts) {
|
||||
console.log(` Scale: ${count.toLocaleString()} observations...`);
|
||||
const observations = generateScaleDataset(count);
|
||||
const sessionCount = new Set(observations.map(o => o.sessionId)).size;
|
||||
|
||||
const heapBefore = process.memoryUsage().heapUsed;
|
||||
|
||||
const buildStart = performance.now();
|
||||
const bm25 = new SearchIndex();
|
||||
const vector = new VectorIndex();
|
||||
const kv = mockKV();
|
||||
const dims = 384;
|
||||
|
||||
for (const obs of observations) {
|
||||
bm25.add(obs);
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts].join(" ");
|
||||
vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
|
||||
await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
|
||||
}
|
||||
const buildMs = performance.now() - buildStart;
|
||||
|
||||
const heapAfter = process.memoryUsage().heapUsed;
|
||||
|
||||
const mockEmbed: any = {
|
||||
name: "deterministic", dimensions: dims,
|
||||
embed: async (t: string) => deterministicEmbedding(t, dims),
|
||||
embedBatch: async (ts: string[]) => ts.map(t => deterministicEmbedding(t, dims)),
|
||||
};
|
||||
const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
|
||||
|
||||
let bm25Total = 0;
|
||||
let hybridTotal = 0;
|
||||
const iters = 20;
|
||||
|
||||
for (let i = 0; i < iters; i++) {
|
||||
const q = SEARCH_QUERIES[i % SEARCH_QUERIES.length];
|
||||
const s1 = performance.now();
|
||||
bm25.search(q, 10);
|
||||
bm25Total += performance.now() - s1;
|
||||
|
||||
const s2 = performance.now();
|
||||
await hybrid.search(q, 10);
|
||||
hybridTotal += performance.now() - s2;
|
||||
}
|
||||
|
||||
const bm25Ser = bm25.serialize();
|
||||
const vecSer = vector.serialize();
|
||||
|
||||
const allText = observations.map(o =>
|
||||
`- ${o.title}: ${o.narrative.slice(0, 80)}... [${o.concepts.slice(0, 3).join(", ")}]`
|
||||
).join("\n");
|
||||
const builtinTokens = estimateTokens(allText);
|
||||
|
||||
const truncatedText = observations.slice(0, 200).map(o =>
|
||||
`- ${o.title}: ${o.narrative.slice(0, 60)}... [${o.concepts.slice(0, 3).join(", ")}]`
|
||||
).join("\n");
|
||||
const builtin200Tokens = estimateTokens(truncatedText);
|
||||
|
||||
let totalResultTokens = 0;
|
||||
for (let i = 0; i < iters; i++) {
|
||||
const q = SEARCH_QUERIES[i % SEARCH_QUERIES.length];
|
||||
const results = await hybrid.search(q, 10);
|
||||
totalResultTokens += estimateTokens(JSON.stringify(results.map(r => r.observation)));
|
||||
}
|
||||
const agentmemoryTokens = Math.round(totalResultTokens / iters);
|
||||
|
||||
results.push({
|
||||
scale: count,
|
||||
sessions: sessionCount,
|
||||
index_build_ms: Math.round(buildMs),
|
||||
index_build_per_doc_ms: +(buildMs / count).toFixed(3),
|
||||
bm25_search_ms: +(bm25Total / iters).toFixed(3),
|
||||
hybrid_search_ms: +(hybridTotal / iters).toFixed(3),
|
||||
index_size_kb: Math.round(Buffer.byteLength(bm25Ser, "utf-8") / 1024),
|
||||
vector_size_kb: Math.round(Buffer.byteLength(vecSer, "utf-8") / 1024),
|
||||
heap_mb: Math.round((heapAfter - heapBefore) / 1024 / 1024),
|
||||
builtin_tokens: builtinTokens,
|
||||
builtin_200line_tokens: builtin200Tokens,
|
||||
agentmemory_tokens: agentmemoryTokens,
|
||||
token_savings_pct: Math.round((1 - agentmemoryTokens / builtinTokens) * 100),
|
||||
builtin_unreachable_pct: count <= 200 ? 0 : Math.round((1 - 200 / count) * 100),
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
async function benchmarkCrossSession(): Promise<CrossSessionResult[]> {
|
||||
const { observations } = generateDataset();
|
||||
const results: CrossSessionResult[] = [];
|
||||
|
||||
const bm25 = new SearchIndex();
|
||||
const kv = mockKV();
|
||||
const vector = new VectorIndex();
|
||||
const dims = 384;
|
||||
|
||||
for (const obs of observations) {
|
||||
bm25.add(obs);
|
||||
const text = [obs.title, obs.narrative, ...obs.concepts].join(" ");
|
||||
vector.add(obs.id, obs.sessionId, deterministicEmbedding(text, dims));
|
||||
await kv.set(`mem:obs:${obs.sessionId}`, obs.id, obs);
|
||||
}
|
||||
|
||||
const mockEmbed: any = {
|
||||
name: "deterministic", dimensions: dims,
|
||||
embed: async (t: string) => deterministicEmbedding(t, dims),
|
||||
embedBatch: async (ts: string[]) => ts.map(t => deterministicEmbedding(t, dims)),
|
||||
};
|
||||
const hybrid = new HybridSearch(bm25, vector, mockEmbed, kv as never, 0.4, 0.6, 0);
|
||||
|
||||
const crossQueries: Array<{
|
||||
query: string;
|
||||
targetConcepts: string[];
|
||||
targetSessionRange: [number, number];
|
||||
currentSession: number;
|
||||
}> = [
|
||||
{ query: "How did we set up OAuth providers?", targetConcepts: ["oauth", "nextauth"], targetSessionRange: [5, 9], currentSession: 29 },
|
||||
{ query: "What was the N+1 query fix?", targetConcepts: ["n+1", "eager-loading"], targetSessionRange: [10, 14], currentSession: 28 },
|
||||
{ query: "PostgreSQL full-text search setup", targetConcepts: ["full-text-search", "tsvector"], targetSessionRange: [10, 14], currentSession: 27 },
|
||||
{ query: "bcrypt password hashing configuration", targetConcepts: ["bcrypt", "password-hashing"], targetSessionRange: [5, 9], currentSession: 25 },
|
||||
{ query: "Vitest unit testing setup", targetConcepts: ["vitest", "unit-testing"], targetSessionRange: [20, 24], currentSession: 29 },
|
||||
{ query: "webhook retry exponential backoff", targetConcepts: ["webhooks", "exponential-backoff"], targetSessionRange: [15, 19], currentSession: 29 },
|
||||
{ query: "ESLint flat config migration", targetConcepts: ["eslint", "linting"], targetSessionRange: [0, 4], currentSession: 29 },
|
||||
{ query: "Kubernetes HPA autoscaling configuration", targetConcepts: ["hpa", "autoscaling", "kubernetes"], targetSessionRange: [25, 29], currentSession: 29 },
|
||||
{ query: "Prisma database seed script", targetConcepts: ["seeding", "faker", "prisma"], targetSessionRange: [10, 14], currentSession: 26 },
|
||||
{ query: "API cursor-based pagination", targetConcepts: ["cursor-based", "pagination"], targetSessionRange: [15, 19], currentSession: 29 },
|
||||
{ query: "CSRF protection double-submit cookie", targetConcepts: ["csrf", "cookies"], targetSessionRange: [5, 9], currentSession: 29 },
|
||||
{ query: "blue-green deployment rollback", targetConcepts: ["blue-green", "rollback", "zero-downtime"], targetSessionRange: [25, 29], currentSession: 29 },
|
||||
];
|
||||
|
||||
for (const cq of crossQueries) {
|
||||
const targetObs = observations.filter(o =>
|
||||
o.concepts.some(c => cq.targetConcepts.includes(c))
|
||||
);
|
||||
const targetIds = new Set(targetObs.map(o => o.id));
|
||||
|
||||
const start = performance.now();
|
||||
const bm25Results = bm25.search(cq.query, 20);
|
||||
const hybridResults = await hybrid.search(cq.query, 20);
|
||||
const latency = performance.now() - start;
|
||||
|
||||
const bm25Rank = bm25Results.findIndex(r => targetIds.has(r.obsId));
|
||||
const hybridRank = hybridResults.findIndex(r => targetIds.has(r.observation.id));
|
||||
|
||||
const builtinLines = 200;
|
||||
const visibleObs = observations.slice(0, builtinLines);
|
||||
const builtinFound = visibleObs.some(o => targetIds.has(o.id));
|
||||
|
||||
const sessionsApart = cq.currentSession - cq.targetSessionRange[0];
|
||||
|
||||
results.push({
|
||||
query: cq.query,
|
||||
target_session: `ses_${cq.targetSessionRange[0].toString().padStart(3, "0")}-${cq.targetSessionRange[1].toString().padStart(3, "0")}`,
|
||||
current_session: `ses_${cq.currentSession.toString().padStart(3, "0")}`,
|
||||
sessions_apart: sessionsApart,
|
||||
bm25_found: bm25Rank >= 0,
|
||||
bm25_rank: bm25Rank >= 0 ? bm25Rank + 1 : -1,
|
||||
hybrid_found: hybridRank >= 0,
|
||||
hybrid_rank: hybridRank >= 0 ? hybridRank + 1 : -1,
|
||||
builtin_found: builtinFound,
|
||||
latency_ms: latency,
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
function generateReport(scale: ScaleResult[], cross: CrossSessionResult[]): string {
|
||||
const lines: string[] = [];
|
||||
const w = (s: string) => lines.push(s);
|
||||
|
||||
w("# agentmemory v0.6.0 — Scale & Cross-Session Evaluation");
|
||||
w("");
|
||||
w(`**Date:** ${new Date().toISOString()}`);
|
||||
w(`**Platform:** ${process.platform} ${process.arch}, Node ${process.version}`);
|
||||
w("");
|
||||
|
||||
w("## 1. Scale: agentmemory vs Built-in Memory");
|
||||
w("");
|
||||
w("Every built-in agent memory (CLAUDE.md, .cursorrules, Cline's memory-bank) loads ALL memory into context every session. agentmemory searches and returns only relevant results.");
|
||||
w("");
|
||||
w("| Observations | Sessions | Index Build | BM25 Search | Hybrid Search | Heap | Context Tokens (built-in) | Context Tokens (agentmemory) | Savings | Built-in Unreachable |");
|
||||
w("|-------------|----------|------------|-------------|---------------|------|--------------------------|-----------------------------|---------|--------------------|");
|
||||
|
||||
for (const r of scale) {
|
||||
w(`| ${r.scale.toLocaleString()} | ${r.sessions} | ${r.index_build_ms}ms | ${r.bm25_search_ms}ms | ${r.hybrid_search_ms}ms | ${r.heap_mb}MB | ${r.builtin_tokens.toLocaleString()} | ${r.agentmemory_tokens.toLocaleString()} | ${r.token_savings_pct}% | ${r.builtin_unreachable_pct}% |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("### What the numbers mean");
|
||||
w("");
|
||||
w("**Context Tokens (built-in):** How many tokens Claude Code/Cursor/Cline would consume loading ALL memory into the context window. At 5,000 observations, this is ~250K tokens — exceeding most context windows entirely.");
|
||||
w("");
|
||||
w("**Context Tokens (agentmemory):** How many tokens the top-10 search results consume. Stays constant regardless of corpus size.");
|
||||
w("");
|
||||
w("**Built-in Unreachable:** Percentage of memories that built-in systems CANNOT access because they exceed the 200-line MEMORY.md cap or context window limits. At 1,000 observations, 80% of your project history is invisible.");
|
||||
w("");
|
||||
|
||||
w("### Storage Costs");
|
||||
w("");
|
||||
w("| Observations | BM25 Index | Vector Index (d=384) | Total Storage |");
|
||||
w("|-------------|-----------|---------------------|---------------|");
|
||||
for (const r of scale) {
|
||||
const total = r.index_size_kb + r.vector_size_kb;
|
||||
w(`| ${r.scale.toLocaleString()} | ${r.index_size_kb.toLocaleString()} KB | ${r.vector_size_kb.toLocaleString()} KB | ${(total / 1024).toFixed(1)} MB |`);
|
||||
}
|
||||
|
||||
w("");
|
||||
w("## 2. Cross-Session Retrieval");
|
||||
w("");
|
||||
w("Can the system find relevant information from past sessions? This is impossible for built-in memory once observations exceed the line/context cap.");
|
||||
w("");
|
||||
w("| Query | Target Session | Gap | BM25 Found | BM25 Rank | Hybrid Found | Hybrid Rank | Built-in Visible |");
|
||||
w("|-------|---------------|-----|-----------|-----------|-------------|-------------|-----------------|");
|
||||
|
||||
for (const r of cross) {
|
||||
w(`| ${r.query.slice(0, 40)}${r.query.length > 40 ? "..." : ""} | ${r.target_session} | ${r.sessions_apart} | ${r.bm25_found ? "Yes" : "No"} | ${r.bm25_rank > 0 ? `#${r.bm25_rank}` : "-"} | ${r.hybrid_found ? "Yes" : "No"} | ${r.hybrid_rank > 0 ? `#${r.hybrid_rank}` : "-"} | ${r.builtin_found ? "Yes" : "No"} |`);
|
||||
}
|
||||
|
||||
const bm25Found = cross.filter(r => r.bm25_found).length;
|
||||
const hybridFound = cross.filter(r => r.hybrid_found).length;
|
||||
const builtinFound = cross.filter(r => r.builtin_found).length;
|
||||
|
||||
w("");
|
||||
w(`**Summary:** agentmemory BM25 found ${bm25Found}/${cross.length} cross-session queries. Hybrid found ${hybridFound}/${cross.length}. Built-in memory (200-line cap) could only reach ${builtinFound}/${cross.length}.`);
|
||||
|
||||
w("");
|
||||
w("## 3. The Context Window Problem");
|
||||
w("");
|
||||
w("```");
|
||||
w("Agent context window: ~200K tokens");
|
||||
w("System prompt + tools: ~20K tokens");
|
||||
w("User conversation: ~30K tokens");
|
||||
w("Available for memory: ~150K tokens");
|
||||
w("");
|
||||
w("At 50 tokens/observation:");
|
||||
w(" 200 observations = 10,000 tokens (fits, but 200-line cap hits first)");
|
||||
w(" 1,000 observations = 50,000 tokens (33% of available budget)");
|
||||
w(" 5,000 observations = 250,000 tokens (EXCEEDS total context window)");
|
||||
w("");
|
||||
w("agentmemory top-10 results:");
|
||||
w(` Any corpus size = ~${scale[0]?.agentmemory_tokens.toLocaleString() || "500"} tokens (0.3% of budget)`);
|
||||
w("```");
|
||||
w("");
|
||||
|
||||
w("## 4. What Built-in Memory Cannot Do");
|
||||
w("");
|
||||
w("| Capability | Built-in (CLAUDE.md) | agentmemory |");
|
||||
w("|-----------|---------------------|-------------|");
|
||||
w("| Semantic search | No (keyword grep only) | BM25 + vector + graph |");
|
||||
w("| Scale beyond 200 lines | No (hard cap) | Unlimited |");
|
||||
w("| Cross-session recall | Only if in 200-line window | Full corpus search |");
|
||||
w("| Cross-agent sharing | No (per-agent files) | MCP + REST API |");
|
||||
w("| Multi-agent coordination | No | Leases, signals, actions |");
|
||||
w("| Temporal queries | No | Point-in-time graph |");
|
||||
w("| Memory lifecycle | No (manual pruning) | Ebbinghaus decay + eviction |");
|
||||
w("| Knowledge graph | No | Entity extraction + traversal |");
|
||||
w("| Query expansion | No | LLM-generated reformulations |");
|
||||
w("| Retention scoring | No | Time-frequency decay model |");
|
||||
w("| Real-time dashboard | No (read files manually) | Viewer on :3113 |");
|
||||
w("| Concurrent access | No (file lock) | Keyed mutex + KV store |");
|
||||
w("");
|
||||
|
||||
w("## 5. When to Use What");
|
||||
w("");
|
||||
w("**Use built-in memory (CLAUDE.md) when:**");
|
||||
w("- You have < 200 items to remember");
|
||||
w("- Single agent, single project");
|
||||
w("- Preferences and quick facts only");
|
||||
w("- Zero setup is the priority");
|
||||
w("");
|
||||
w("**Use agentmemory when:**");
|
||||
w("- Project history exceeds 200 observations");
|
||||
w("- You need to recall specific incidents from weeks ago");
|
||||
w("- Multiple agents work on the same codebase");
|
||||
w("- You want semantic search (\"how does auth work?\") not just keyword matching");
|
||||
w("- You need to track memory quality, decay, and lifecycle");
|
||||
w("- You want a shared memory layer across Claude Code, Cursor, Windsurf, etc.");
|
||||
w("");
|
||||
w("Built-in memory is your sticky notes. agentmemory is the searchable database behind them.");
|
||||
w("");
|
||||
|
||||
w("---");
|
||||
w(`*Scale tests: ${scale.length} corpus sizes. Cross-session tests: ${cross.length} queries targeting specific past sessions.*`);
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log("=== agentmemory Scale & Cross-Session Evaluation ===\n");
|
||||
|
||||
console.log("1. Scale benchmarks...");
|
||||
const scaleResults = await benchmarkScale([240, 1_000, 5_000, 10_000, 50_000]);
|
||||
|
||||
console.log("\n2. Cross-session retrieval...");
|
||||
const crossResults = await benchmarkCrossSession();
|
||||
|
||||
console.log("");
|
||||
const report = generateReport(scaleResults, crossResults);
|
||||
writeFileSync("benchmark/SCALE.md", report);
|
||||
console.log(report);
|
||||
console.log(`\nReport written to benchmark/SCALE.md`);
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
Reference in New Issue
Block a user