wehub-resource-sync
158 lines
5.3 KiB
Markdown
158 lines
5.3 KiB
Markdown
# OpenAI Chat‑Completion TTFT Benchmark
|
||
|
||
Measure **time‑to‑first‑token (TTFT)** — and optional cache‑hit latency — from **any
|
||
server that speaks the OpenAI `/v1` API** (vLLM, llama.cpp with `--api`,
|
||
OpenAI‑proxy, etc.).
|
||
|
||
> **Why run it?**
|
||
> • Compare *cold* latency vs. *cache‑hit* latency.
|
||
> • Verify whether a KV‑cache (VRAM, SSD, LMCache, …) actually helps.
|
||
> • Collect JSONL you can plot
|
||
|
||
---
|
||
|
||
## 1 · Prerequisites
|
||
|
||
| Requirement | Notes |
|
||
|-------------|-------|
|
||
| **Running endpoint** | Must expose the OpenAI REST interface (default URL `http://localhost:8000/v1`). |
|
||
|
||
For more information on how to serve an endpoint using vllm and LMCache,
|
||
|
||
---
|
||
|
||
## 2 · Command‑line flags
|
||
|
||
| Flag / shorthand | Default | Meaning |
|
||
|------------------|---------|---------|
|
||
| `--api_base` | `http://localhost:8000/v1` | URL of the OpenAI‑style endpoint. |
|
||
| `--api_key` | `EMPTY` | Any string (ignored by most local servers). |
|
||
| `--model` | *first model from* `/models` | Explicit model ID. |
|
||
| `-C`, `--context_file` | *see table below* | Document inserted before the prompt. |
|
||
| `--max_ctx_tokens` | **131 072** | Upper bound *after* truncation. |
|
||
| `--prompt` | `"Summarize this text"` | Prompt appended after the document. |
|
||
| `--num_following`| **1** | Extra TTFT‑measured requests after the baseline. |
|
||
| `-F`, `--flush_cache` | off | Flush GPU KV‑cache **once** after run 1. |
|
||
| `--out` | `benchmark.jsonl` | JSONL log (cleared at start). |
|
||
|
||
### Behaviour of `--context_file`
|
||
|
||
| Invocation | Document used |
|
||
|------------|---------------|
|
||
| *(flag omitted)* | Synthetic ASCII filler based on max ctx length input|
|
||
| `--context_file` *(no path)* | Bundled `ffmpeg.txt` (one dir up) |
|
||
| `--context_file /path/doc.txt` | Exact file you specify |
|
||
|
||
> **Legacy shorthand** – you may also run
|
||
> `python openai_chat_completion_client.py <PORT>`
|
||
> and every other option remains default.
|
||
|
||
---
|
||
|
||
## 3 · Quick start
|
||
|
||
Cold + warm measurement (two requests total):
|
||
|
||
```bash
|
||
python openai_chat_completion_client.py --num_following 1
|
||
```
|
||
|
||
Example console output
|
||
|
||
```
|
||
=== Run 1: baseline TTFT ===
|
||
TTFT_1 = 0.429s
|
||
(no KV‑cache flush requested)
|
||
|
||
=== Run 2: TTFT continued ===
|
||
TTFT_2 = 0.081s
|
||
```
|
||
|
||
`benchmark.jsonl`
|
||
|
||
```json
|
||
{"run_index":1,"context_tokens":120938,"ttft_seconds":0.429}
|
||
{"run_index":2,"context_tokens":120938,"ttft_seconds":0.081}
|
||
```
|
||
|
||
---
|
||
|
||
## 4 · Advanced use
|
||
|
||
### 4.1 · Benchmark after cache eviction
|
||
|
||
```bash
|
||
python openai_chat_completion_client.py \
|
||
-C war_and_peace.txt \
|
||
--num_following 3 \
|
||
--flush_cache \
|
||
--prompt "Give me a concise outline." \
|
||
--out warpeace_flush.jsonl
|
||
```
|
||
|
||
* Run 1 – cold
|
||
* Cache flushed
|
||
* Run 2 – cold again (miss)
|
||
* Runs 3‑4 – warm (hits)
|
||
|
||
### 4.2 · Stress maximum context
|
||
|
||
```bash
|
||
python openai_chat_completion_client.py \
|
||
--max_ctx_tokens 131072 \
|
||
--num_following 1 -F
|
||
```
|
||
|
||
Generates a k‑char filler, truncates to fit
|
||
`≤ max_ctx ` tokens (keeps a **2 048‑token safety margin**), then
|
||
measures cold vs. warm TTFT.
|
||
|
||
---
|
||
|
||
## 5 · Output schema
|
||
|
||
Each JSONL line contains:
|
||
|
||
| Key | Type | Description |
|
||
|-----|------|-------------|
|
||
| `run_index` | int | 1 = baseline, 2… = follow‑ups |
|
||
| `context_tokens` | int | Tokens after truncation |
|
||
| `ttft_seconds` | float | Wall‑clock seconds to **first** streamed token |
|
||
|
||
Concatenate multiple logs with `cat` and plot as you like.
|
||
|
||
---
|
||
|
||
## 6 · Implementation notes
|
||
|
||
* **Safety margin** – `SAFETY_MARGIN = 2048` tokens so the request never
|
||
overruns model context even on tokenizer quirks.
|
||
* **Spinner** – Red arrows animate while waiting for token #1, stop instantly
|
||
on arrival for visual TTFT confirmation.
|
||
* **Tokenizer fallback** – If the matching tokenizer can’t load, the script
|
||
degrades to the heuristic “≈ 4 chars = 1 token”.
|
||
* **Cache‑flush routine** – Sends ten *1‑token* completions built on a
|
||
100 k‑char filler doc to evict KV blocks from VRAM.
|
||
|
||
## 7 · Batch driver script (`bench_ttft_sweep.sh`)
|
||
|
||
This is an example basic bash script you might use to do a sweep across different context lengths, combining results to one file for easy comparison of caching methods.
|
||
|
||
|
||
|
||
### What the script does
|
||
|
||
| Step | Detail |
|
||
|------|--------|
|
||
| **1. Configure variables** | `BENCH` points to the Python benchmark, `MASTER_OUT` is the cumulative log, and `CONTEXT_SIZES` lists the target document lengths (in **tokens**). |
|
||
| **2. Per‑size run** | For each length the script launches the benchmark with:<br>• custom `--max_ctx_tokens` (see above)<br>• one cache‑hit follow‑up (`--num_following 1`)<br>• an explicit **70 B** Llama 3 checkpoint via `--model` |
|
||
| **3. Log collation** | Each invocation writes its own JSONL (`ttft_<N>.jsonl`). Those lines are immediately concatenated into **`all_ttft_results.jsonl`**, producing a tidy file like: <br>`{"run_index":1,"context_tokens":32000,"ttft_seconds":0.45}` |
|
||
| **4. Done banner** | After the loop finishes you get a green check‑mark and the path to the merged log. |
|
||
|
||
#### Customising
|
||
|
||
* **Change the model** — edit `--model …` to point at any endpoint‑visible name.
|
||
* **Different sizes** — just tweak the `CONTEXT_SIZES` array.
|
||
* **More follow‑ups** — bump `--num_following` if you want deeper cache‑hit sampling.
|
||
|