lmcache--lmcache/examples/online_session

OpenAI Chat‑Completion TTFT Benchmark

Measure time‑to‑first‑token (TTFT) — and optional cache‑hit latency — from any server that speaks the OpenAI /v1 API (vLLM, llama.cpp with --api, OpenAI‑proxy, etc.).

Why run it?
• Compare cold latency vs. cache‑hit latency.
• Verify whether a KV‑cache (VRAM, SSD, LMCache, …) actually helps.
• Collect JSONL you can plot

---

1 · Prerequisites

Requirement	Notes
Running endpoint	Must expose the OpenAI REST interface (default URL http://localhost:8000/v1).

For more information on how to serve an endpoint using vllm and LMCache,

---

2 · Command‑line flags

Flag / shorthand	Default	Meaning
--api_base	http://localhost:8000/v1	URL of the OpenAI‑style endpoint.
--api_key	EMPTY	Any string (ignored by most local servers).
--model	first model from /models	Explicit model ID.
-C, --context_file	see table below	Document inserted before the prompt.
--max_ctx_tokens	131 072	Upper bound after truncation.
--prompt	"Summarize this text"	Prompt appended after the document.
--num_following	1	Extra TTFT‑measured requests after the baseline.
-F, --flush_cache	off	Flush GPU KV‑cache once after run 1.
--out	benchmark.jsonl	JSONL log (cleared at start).

Behaviour of --context_file

Invocation	Document used
(flag omitted)	Synthetic ASCII filler based on max ctx length input
--context_file (no path)	Bundled ffmpeg.txt (one dir up)
--context_file /path/doc.txt	Exact file you specify

Legacy shorthand – you may also run
python openai_chat_completion_client.py <PORT>
and every other option remains default.

---

3 · Quick start

Cold + warm measurement (two requests total):

python openai_chat_completion_client.py --num_following 1

Example console output

=== Run 1: baseline TTFT ===
TTFT_1 = 0.429s
(no KV‑cache flush requested)

=== Run 2: TTFT continued ===
TTFT_2 = 0.081s

benchmark.jsonl

{"run_index":1,"context_tokens":120938,"ttft_seconds":0.429}
{"run_index":2,"context_tokens":120938,"ttft_seconds":0.081}

---

4 · Advanced use

4.1 · Benchmark after cache eviction

python openai_chat_completion_client.py \
  -C war_and_peace.txt        \
  --num_following 3           \
  --flush_cache               \
  --prompt "Give me a concise outline." \
  --out warpeace_flush.jsonl

• Run 1 – cold
• Cache flushed
• Run 2 – cold again (miss)
• Runs 3‑4 – warm (hits)

4.2 · Stress maximum context

python openai_chat_completion_client.py \
  --max_ctx_tokens 131072 \
  --num_following 1 -F

Generates a k‑char filler, truncates to fit ≤ max_ctx  tokens (keeps a 2 048‑token safety margin), then measures cold vs. warm TTFT.

---

5 · Output schema

Each JSONL line contains:

Key	Type	Description
run_index	int	1 = baseline, 2… = follow‑ups
context_tokens	int	Tokens after truncation
ttft_seconds	float	Wall‑clock seconds to first streamed token

Concatenate multiple logs with cat and plot as you like.

---

6 · Implementation notes

• Safety margin – SAFETY_MARGIN = 2048 tokens so the request never overruns model context even on tokenizer quirks.
• Spinner – Red arrows animate while waiting for token #1, stop instantly on arrival for visual TTFT confirmation.
• Tokenizer fallback – If the matching tokenizer can’t load, the script degrades to the heuristic “≈ 4 chars = 1 token”.
• Cache‑flush routine – Sends ten 1‑token completions built on a 100 k‑char filler doc to evict KV blocks from VRAM.

7 · Batch driver script (bench_ttft_sweep.sh)

This is an example basic bash script you might use to do a sweep across different context lengths, combining results to one file for easy comparison of caching methods.

What the script does

Step	Detail
1. Configure variables	BENCH points to the Python benchmark, MASTER_OUT is the cumulative log, and CONTEXT_SIZES lists the target document lengths (in tokens).
2. Per‑size run	For each length the script launches the benchmark with: • custom --max_ctx_tokens (see above) • one cache‑hit follow‑up (--num_following 1) • an explicit 70 B Llama 3 checkpoint via --model
3. Log collation	Each invocation writes its own JSONL (ttft_<N>.jsonl). Those lines are immediately concatenated into all_ttft_results.jsonl, producing a tidy file like: {"run_index":1,"context_tokens":32000,"ttft_seconds":0.45}
4. Done banner	After the loop finishes you get a green check‑mark and the path to the merged log.

Customising

• Change the model — edit --model … to point at any endpoint‑visible name.
• Different sizes — just tweak the CONTEXT_SIZES array.
• More follow‑ups — bump --num_following if you want deeper cache‑hit sampling.