chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
# benchmark/
|
||||
|
||||
Two kinds of numbers live in this directory:
|
||||
|
||||
1. **Quality / retrieval** — `longmemeval-bench.ts`, `quality-eval.ts`,
|
||||
`real-embeddings-eval.ts`, `scale-eval.ts`. Recall, precision, token
|
||||
savings. Documented in `LONGMEMEVAL.md`, `QUALITY.md`,
|
||||
`REAL-EMBEDDINGS.md`, `SCALE.md`.
|
||||
|
||||
2. **Load shape** — `load-100k.ts`. p50 / p90 / p99 latency and
|
||||
throughput against a running daemon. This is the file you want when
|
||||
somebody asks "what's p99 at 100k memories under concurrency 100?".
|
||||
|
||||
## load-100k.ts
|
||||
|
||||
Hand-rolled, dependency-free load harness. Issues real HTTP against a
|
||||
local agentmemory daemon at `http://localhost:3111`, records per-request
|
||||
latency with `performance.now()`, and writes a JSON report per run.
|
||||
|
||||
### What it measures
|
||||
|
||||
For each cell in the matrix `(N, concurrency, endpoint)` it records:
|
||||
|
||||
- `p50_ms`, `p90_ms`, `p99_ms` — nearest-rank percentiles.
|
||||
- `min_ms`, `max_ms`, `ops`, `errors`.
|
||||
- `throughput_per_sec` — wall-clock ops / sec for that cell.
|
||||
|
||||
Default matrix:
|
||||
|
||||
- `N` ∈ {1000, 10000, 100000} — number of memories seeded before the
|
||||
cell runs.
|
||||
- `C` ∈ {1, 10, 100} — concurrent in-flight requests during the cell.
|
||||
- Endpoints under test:
|
||||
- `POST /agentmemory/remember`
|
||||
- `POST /agentmemory/smart-search`
|
||||
- `GET /agentmemory/memories?latest=true`
|
||||
|
||||
Each cell issues `BENCH_OPS=200` requests by default — enough samples
|
||||
for stable p99 without dragging a 100k-seed run past tens of minutes.
|
||||
|
||||
### Why p99 is the number that matters
|
||||
|
||||
p50 tells you the median request feels fast. p90 tells you the bulk of
|
||||
requests feel fast. **p99 tells you the request your tail user hits when
|
||||
they really need it feels fast.** Capacity planning lives here — if you
|
||||
want to size a fleet, scale your daemon, or set an SLO, p99 is the
|
||||
number to plan against. p50 will lie to you.
|
||||
|
||||
### Running it
|
||||
|
||||
```bash
|
||||
# 1. Start the daemon however you normally do (npx, Docker, etc.)
|
||||
npx @agentmemory/agentmemory
|
||||
|
||||
# 2. From the repo root, in another shell:
|
||||
npm run bench:load
|
||||
```
|
||||
|
||||
To override the matrix:
|
||||
|
||||
```bash
|
||||
BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
|
||||
```
|
||||
|
||||
To have the harness spawn a daemon for the run (after `npm run build`):
|
||||
|
||||
```bash
|
||||
AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
|
||||
```
|
||||
|
||||
Other env knobs (see the file header for the canonical list):
|
||||
|
||||
- `AGENTMEMORY_URL` — base URL of the daemon (default
|
||||
`http://localhost:3111`).
|
||||
- `BENCH_SEED` — seed for the `mulberry32` content RNG. Same seed +
|
||||
same daemon build = byte-identical seed corpus.
|
||||
- `BENCH_OUT_DIR` — where the JSON report lands (default
|
||||
`benchmark/results/`).
|
||||
|
||||
### Where results land
|
||||
|
||||
`benchmark/results/load-100k-<short-git-sha>.json`. The harness
|
||||
`mkdir -p`s the directory. The file has a `schema_version: 1` field so
|
||||
future format changes don't silently break consumers.
|
||||
|
||||
### Content generation is seedable
|
||||
|
||||
Synthetic memory content is built from a small noun / verb / concept
|
||||
vocabulary fed by a `mulberry32(BENCH_SEED)` PRNG. Same seed + same
|
||||
build = same corpus. The point isn't "realistic" content (there isn't
|
||||
one realistic content); the point is **reproducibility** — re-running
|
||||
the harness against the same git sha should give the same content
|
||||
mixture going in, so latency variance comes from the daemon and not
|
||||
from JSON payload jitter.
|
||||
|
||||
### Publishing numbers per release
|
||||
|
||||
The release flow appends a `## Performance` section to `CHANGELOG.md`
|
||||
referencing the JSON in `benchmark/results/` for that release's git
|
||||
sha. p99 is the headline number; the JSON is the receipt.
|
||||
Reference in New Issue
Block a user