wehub-resource-sync
101 lines
3.4 KiB
Markdown
101 lines
3.4 KiB
Markdown
# benchmark/
|
|
|
|
Two kinds of numbers live in this directory:
|
|
|
|
1. **Quality / retrieval** — `longmemeval-bench.ts`, `quality-eval.ts`,
|
|
`real-embeddings-eval.ts`, `scale-eval.ts`. Recall, precision, token
|
|
savings. Documented in `LONGMEMEVAL.md`, `QUALITY.md`,
|
|
`REAL-EMBEDDINGS.md`, `SCALE.md`.
|
|
|
|
2. **Load shape** — `load-100k.ts`. p50 / p90 / p99 latency and
|
|
throughput against a running daemon. This is the file you want when
|
|
somebody asks "what's p99 at 100k memories under concurrency 100?".
|
|
|
|
## load-100k.ts
|
|
|
|
Hand-rolled, dependency-free load harness. Issues real HTTP against a
|
|
local agentmemory daemon at `http://localhost:3111`, records per-request
|
|
latency with `performance.now()`, and writes a JSON report per run.
|
|
|
|
### What it measures
|
|
|
|
For each cell in the matrix `(N, concurrency, endpoint)` it records:
|
|
|
|
- `p50_ms`, `p90_ms`, `p99_ms` — nearest-rank percentiles.
|
|
- `min_ms`, `max_ms`, `ops`, `errors`.
|
|
- `throughput_per_sec` — wall-clock ops / sec for that cell.
|
|
|
|
Default matrix:
|
|
|
|
- `N` ∈ {1000, 10000, 100000} — number of memories seeded before the
|
|
cell runs.
|
|
- `C` ∈ {1, 10, 100} — concurrent in-flight requests during the cell.
|
|
- Endpoints under test:
|
|
- `POST /agentmemory/remember`
|
|
- `POST /agentmemory/smart-search`
|
|
- `GET /agentmemory/memories?latest=true`
|
|
|
|
Each cell issues `BENCH_OPS=200` requests by default — enough samples
|
|
for stable p99 without dragging a 100k-seed run past tens of minutes.
|
|
|
|
### Why p99 is the number that matters
|
|
|
|
p50 tells you the median request feels fast. p90 tells you the bulk of
|
|
requests feel fast. **p99 tells you the request your tail user hits when
|
|
they really need it feels fast.** Capacity planning lives here — if you
|
|
want to size a fleet, scale your daemon, or set an SLO, p99 is the
|
|
number to plan against. p50 will lie to you.
|
|
|
|
### Running it
|
|
|
|
```bash
|
|
# 1. Start the daemon however you normally do (npx, Docker, etc.)
|
|
npx @agentmemory/agentmemory
|
|
|
|
# 2. From the repo root, in another shell:
|
|
npm run bench:load
|
|
```
|
|
|
|
To override the matrix:
|
|
|
|
```bash
|
|
BENCH_N=1000 BENCH_C=1,10 BENCH_OPS=100 npm run bench:load
|
|
```
|
|
|
|
To have the harness spawn a daemon for the run (after `npm run build`):
|
|
|
|
```bash
|
|
AGENTMEMORY_BENCH_AUTOSTART=1 npm run bench:load
|
|
```
|
|
|
|
Other env knobs (see the file header for the canonical list):
|
|
|
|
- `AGENTMEMORY_URL` — base URL of the daemon (default
|
|
`http://localhost:3111`).
|
|
- `BENCH_SEED` — seed for the `mulberry32` content RNG. Same seed +
|
|
same daemon build = byte-identical seed corpus.
|
|
- `BENCH_OUT_DIR` — where the JSON report lands (default
|
|
`benchmark/results/`).
|
|
|
|
### Where results land
|
|
|
|
`benchmark/results/load-100k-<short-git-sha>.json`. The harness
|
|
`mkdir -p`s the directory. The file has a `schema_version: 1` field so
|
|
future format changes don't silently break consumers.
|
|
|
|
### Content generation is seedable
|
|
|
|
Synthetic memory content is built from a small noun / verb / concept
|
|
vocabulary fed by a `mulberry32(BENCH_SEED)` PRNG. Same seed + same
|
|
build = same corpus. The point isn't "realistic" content (there isn't
|
|
one realistic content); the point is **reproducibility** — re-running
|
|
the harness against the same git sha should give the same content
|
|
mixture going in, so latency variance comes from the daemon and not
|
|
from JSON payload jitter.
|
|
|
|
### Publishing numbers per release
|
|
|
|
The release flow appends a `## Performance` section to `CHANGELOG.md`
|
|
referencing the JSON in `benchmark/results/` for that release's git
|
|
sha. p99 is the headline number; the JSON is the receipt.
|