chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,196 @@
|
||||
# Multi-Turn LLM Benchmark
|
||||
|
||||
A benchmark tool for OpenAI-compatible LLM inference servers that supports
|
||||
multi-turn conversations with configurable prefix cache hit rates, input/output
|
||||
sequence lengths, and cross-session prefix sharing.
|
||||
|
||||
## Entry Point
|
||||
|
||||
```
|
||||
python -m ray.llm._internal.serve.benchmark.cli [OPTIONS]
|
||||
```
|
||||
|
||||
## Modes
|
||||
|
||||
| Command | Mode | Description |
|
||||
|---------|------|-------------|
|
||||
| `... -s` | Smoke | Single request health check |
|
||||
| `... --concurrency 8 ...` | Direct (concurrency) | Closed-loop concurrency benchmark |
|
||||
| `... --request-rate 10 ...` | Direct (rate) | Constant-QPS benchmark |
|
||||
| `... -i` | Interactive server | Long-running server with UNIX socket control |
|
||||
| `... -i --client` | Interactive client | Connect to server; REPL or `--cmd` one-shot |
|
||||
|
||||
## Quick Examples
|
||||
|
||||
### Smoke test
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli -s \
|
||||
-u http://localhost:8000 -m my-model
|
||||
```
|
||||
|
||||
### Concurrency benchmark
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli \
|
||||
-u http://localhost:8000 -m meta-llama/Llama-3-8B-Instruct \
|
||||
--concurrency 8 --num-sessions 200 \
|
||||
--isl 2000 --osl 200 --hit-rate 0.85 --num-turns 5 \
|
||||
--think-time 1.0 --save-result results.json
|
||||
```
|
||||
|
||||
### Rate benchmark
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli \
|
||||
-u http://localhost:8000 -m meta-llama/Llama-3-8B-Instruct \
|
||||
--request-rate 10 --duration 120 \
|
||||
--isl 2000 --osl 200 --hit-rate 0.85 --num-turns 5 \
|
||||
--warm-up 10 --save-result results.json
|
||||
```
|
||||
|
||||
### Interactive server
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i \
|
||||
-u http://localhost:8000 -m meta-llama/Llama-3-8B-Instruct \
|
||||
--isl 2000 --osl 200 --hit-rate 0.85 --num-turns 5
|
||||
```
|
||||
|
||||
### Interactive client (REPL)
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i --client
|
||||
```
|
||||
|
||||
### Interactive client (one-shot)
|
||||
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i --client --cmd "rate 10"
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i --client --cmd "status"
|
||||
```
|
||||
|
||||
## Workload Parameters
|
||||
|
||||
All workload parameters use **simple mode**: you specify user-facing values
|
||||
and the tool derives internal parameters (per-turn user tokens `u` and
|
||||
system prompt tokens `s`) automatically.
|
||||
|
||||
| Parameter | Flag | Description |
|
||||
|-----------|------|-------------|
|
||||
| ISL | `--isl` | Average input sequence length (tokens) across all turns |
|
||||
| OSL | `--osl` | Output tokens per turn |
|
||||
| Hit rate | `--hit-rate` | Target prefix cache hit rate [0, 1] |
|
||||
| Shared system prompt ratio | `--shared-system-prompt-ratio` | Fraction of system prompt shared across sessions (default: 0.0) |
|
||||
| Num turns | `--num-turns` | Number of turns per conversation session |
|
||||
| Think time | `--think-time` | Simulated user think-time between turns in seconds (default: 0) |
|
||||
| First chunk threshold | `--first-chunk-threshold` | Number of SSE content chunks before recording first-chunk latency (default: 16) |
|
||||
|
||||
The solver derives `user_tokens` (new user tokens per turn) and `sys_tokens`
|
||||
(total system prompt tokens) from these inputs. The `print_summary()` output
|
||||
shows the resolved per-turn token breakdown including cached vs. new tokens at
|
||||
each turn.
|
||||
|
||||
## Tokenizer
|
||||
|
||||
By default, `--tokenizer` is `None`, which causes the tool to use the
|
||||
`--model` value as the HuggingFace tokenizer name. This works when `--model`
|
||||
is a valid HuggingFace model ID (e.g., `meta-llama/Llama-3-8B-Instruct`).
|
||||
|
||||
Provide `--tokenizer` explicitly when:
|
||||
- The `--model` value is an alias or deployment name that is not a valid
|
||||
HuggingFace repo (e.g., `--model my-deployment --tokenizer meta-llama/Llama-3-8B-Instruct`).
|
||||
- You want to use a local tokenizer path.
|
||||
|
||||
## Warm-Up Strategies
|
||||
|
||||
### Concurrency mode
|
||||
|
||||
Warm-up is **automatic** using entropy-based detection. The tool monitors the
|
||||
distribution of active turns across concurrent sessions. Once the Shannon
|
||||
entropy of the turn distribution reaches 50% of its theoretical maximum, the
|
||||
pool is considered at steady state and measurement begins. All requests
|
||||
dispatched before that point are discarded.
|
||||
|
||||
### Rate mode
|
||||
|
||||
Warm-up is **time-based** via the `--warm-up` flag (in seconds). All requests
|
||||
whose dispatch time falls within the warm-up window are excluded from reported
|
||||
metrics. Set this to allow the server's KV cache to fill and stabilize.
|
||||
|
||||
### Interactive mode
|
||||
|
||||
Warm-up is **manual**. The operator starts traffic with `rate <qps>`, waits
|
||||
for the system to stabilize, then explicitly starts a measurement window with
|
||||
`start` or `measure <n>`.
|
||||
|
||||
## Interactive Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `help` | Show available commands |
|
||||
| `rate <qps>` | Set target request rate (0 to pause) |
|
||||
| `start` | Start open-ended measurement window |
|
||||
| `measure <n>` | Start measurement capturing next `n` completed requests |
|
||||
| `stop` | Stop measurement and print summary |
|
||||
| `status` | Show current state: QPS, inflight, completed, measured |
|
||||
| `workload [k=v ...]` | Show or update workload parameters (e.g., `workload isl=3000 osl=300`) |
|
||||
| `save [path]` | Save last measurement window to JSON |
|
||||
| `save-dir <path>` | Set default directory for saved results |
|
||||
| `quit` | Stop the benchmark server |
|
||||
|
||||
## JSON Result Schema
|
||||
|
||||
Results saved with `--save-result` (direct mode) contain these top-level keys:
|
||||
|
||||
| Key | Description |
|
||||
|-----|-------------|
|
||||
| `config` | Run configuration (concurrency/rate, model, etc.) |
|
||||
| `spec` | Resolved workload spec with per-turn token breakdown |
|
||||
| `first_chunk_threshold` | Number of chunks before recording first-chunk latency |
|
||||
| `benchmark` | Run metadata: total requests, duration, warm-up info |
|
||||
| `stats` | Aggregate latency statistics (avg, P50, P90, P99 for TTFT, FC, TPOT, latency) |
|
||||
| `per_turn` | Per-turn breakdown of count, avg ISL, and latency percentiles |
|
||||
| `raw_metrics` | Array of per-request metrics (session_id, turn, all latency fields, token counts) |
|
||||
|
||||
Interactive mode saves with `save` produce a similar structure with a `window`
|
||||
summary instead of `benchmark`/`stats`/`per_turn`.
|
||||
|
||||
## Typical Workflow
|
||||
|
||||
1. **Smoke test** to verify connectivity:
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli -s -u http://localhost:8000 -m my-model
|
||||
```
|
||||
|
||||
2. **Direct benchmark** for a fixed workload:
|
||||
```bash
|
||||
python -m ray.llm._internal.serve.benchmark.cli \
|
||||
--concurrency 8 --num-sessions 200 \
|
||||
--isl 2000 --osl 200 --hit-rate 0.85 --num-turns 5 \
|
||||
-u http://localhost:8000 -m meta-llama/Llama-3-8B-Instruct \
|
||||
--save-result concurrency_8.json
|
||||
```
|
||||
|
||||
3. **Interactive mode** for exploratory testing:
|
||||
```bash
|
||||
# Terminal 1: start server
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i \
|
||||
--isl 2000 --osl 200 --hit-rate 0.85 --num-turns 5 \
|
||||
-u http://localhost:8000 -m meta-llama/Llama-3-8B-Instruct
|
||||
|
||||
# Terminal 2: control
|
||||
python -m ray.llm._internal.serve.benchmark.cli -i --client
|
||||
benchctl> rate 5
|
||||
benchctl> measure 500
|
||||
benchctl> status
|
||||
benchctl> save results_qps5.json
|
||||
benchctl> rate 10
|
||||
benchctl> measure 500
|
||||
benchctl> save results_qps10.json
|
||||
benchctl> quit
|
||||
```
|
||||
|
||||
4. **Sweep** over multiple configurations: write an external script that loops
|
||||
over the CLI with different parameters. The tool does not include built-in
|
||||
sweep orchestration.
|
||||
Reference in New Issue
Block a user