wehub-resource-sync
153 lines
5.1 KiB
Markdown
153 lines
5.1 KiB
Markdown
# LMCache Observability Example
|
|
|
|
Minimal example showing per-request OTel tracing and metrics for LMCache + vLLM,
|
|
visualized in Grafana.
|
|
|
|
## Stack
|
|
|
|
```
|
|
LMCache / vLLM
|
|
└─ OTLP gRPC → OTel Collector (:4320)
|
|
├─ traces → Tempo (:3200)
|
|
└─ metrics → Prometheus (:9091)
|
|
└─ Grafana (:3000)
|
|
```
|
|
|
|
## Step 1 — Start the observability stack
|
|
|
|
```bash
|
|
cd examples/observability
|
|
docker compose up -d
|
|
```
|
|
|
|
## Step 2 — Start LMCache + vLLM
|
|
|
|
```bash
|
|
MODEL=/your/model/path bash start-server.sh
|
|
```
|
|
|
|
|
|
## Step 3 — Send requests to populate traces
|
|
|
|
```bash
|
|
# Run a short long-doc-qa benchmark: first query is a miss, subsequent
|
|
# queries against the same document are cache hits.
|
|
lmcache bench engine \
|
|
--engine-url http://localhost:8100 \
|
|
--workload long-doc-qa \
|
|
--kv-cache-volume 1 \
|
|
--ldqa-query-per-document 10
|
|
```
|
|
|
|
## Step 4 — Visualize in Grafana
|
|
|
|
Open **http://localhost:3000** → **Explore** → datasource **Tempo**.
|
|
|
|
```
|
|
# All request root spans
|
|
{ name = "request" }
|
|
|
|
# Filter to a specific session
|
|
{ name = "request" && span.session_id = "<request_id>" }
|
|
|
|
# Only cache-hit requests (had a retrieve)
|
|
{ name = "request" } >> { name = "mp.retrieve" }
|
|
|
|
# Requests with less than 50 % cache hit rate
|
|
{ name = "request" && span.hit_rate < 0.5 }
|
|
|
|
# Full cache hits only
|
|
{ name = "request" && span.hit_rate = 1.0 }
|
|
|
|
# Complete misses (lookup ran but nothing was cached)
|
|
{ name = "request" && span.requested_tokens > 0 && span.hit_tokens = 0 }
|
|
```
|
|
|
|
Click any trace to open the waterfall. Each root `request` span carries three
|
|
per-request cache hit rate attributes:
|
|
|
|
| Attribute | Type | Description |
|
|
|-----------|------|-------------|
|
|
| `hit_tokens` | int | tokens served from L1+L2 cache |
|
|
| `requested_tokens` | int | total chunk-aligned tokens submitted for lookup |
|
|
| `hit_rate` | float | `hit_tokens / requested_tokens` (0.0 on a total miss) |
|
|
|
|
```
|
|
request [══════════════════════════════════════] hit_rate=0.75
|
|
mp.lookup_prefetch [════]
|
|
mp.retrieve [════════]
|
|
mp.store [══════]
|
|
```
|
|
|
|
Store-only requests (no lookup phase) do not carry these attributes.
|
|
|
|
The pre-provisioned **LMCache** dashboard under **Dashboards** shows cache hit
|
|
rate, StorageManager read/write rates, and the live trace panel. The collapsed
|
|
**CacheBlend** row adds blend-server panels (see below).
|
|
|
|
## CacheBlend (blend server) traces
|
|
|
|
When LMCache runs the **blend** engine (`lmcache server --engine-type blend`),
|
|
CacheBlend V3 emits its own span tree to Tempo alongside the standard spans.
|
|
Expand the collapsed **CacheBlend** row on the dashboard, or query Tempo:
|
|
|
|
```
|
|
# All CacheBlend request traces
|
|
{ name = "cb.request" }
|
|
|
|
# Requests that actually blended non-prefix (shifted) KV
|
|
{ name = "cb.request" && span.non_prefix_hit_tokens > 0 }
|
|
|
|
# The token-scatter GPU step
|
|
{ name = "cb.scatter" }
|
|
```
|
|
|
|
Click a `cb.request` row to open the waterfall:
|
|
|
|
```
|
|
cb.request
|
|
cb.lookup (attr prefix_chunks; prefix timing is in mp.lookup_prefetch)
|
|
cb.fingerprint_match match probe hashes vs stored fingerprints
|
|
cb.sparse_prefetch non-prefix (shifted) chunks, sparse L2->L1
|
|
(emitted only on an actual L2 load; carries l2_keys)
|
|
cb.retrieve
|
|
cb.scatter L1 -> paged KV per-token slot-scatter + re-RoPE
|
|
cb.store_pre_computed
|
|
cb.store_final
|
|
```
|
|
|
|
The root `cb.request` span carries the V3 hit-rate breakdown
|
|
(`hit_rate = prefix + non-prefix`):
|
|
|
|
| Attribute | Type | Description |
|
|
|-----------|------|-------------|
|
|
| `prefix_hit_tokens` | int | tokens reused from the prefix (L1+L2) |
|
|
| `non_prefix_hit_tokens` | int | tokens reused from sparse non-prefix chunks |
|
|
| `hit_tokens` | int | `prefix_hit_tokens + non_prefix_hit_tokens` |
|
|
| `requested_tokens` | int | total chunk-aligned tokens submitted |
|
|
| `hit_rate` | float | `hit_tokens / requested_tokens` |
|
|
| `prefix_hit_rate` | float | `prefix_hit_tokens / requested_tokens` |
|
|
| `non_prefix_hit_rate` | float | `non_prefix_hit_tokens / requested_tokens` (sums to `hit_rate`) |
|
|
|
|
The **CacheBlend Hit Rate & Chunks** panel overlays the overall token hit rate
|
|
(Prometheus) with the per-request prefix/non-prefix breakdown via
|
|
[TraceQL metrics](https://grafana.com/docs/tempo/latest/metrics-from-traces/),
|
|
served by Tempo's `local-blocks` metrics generator (enabled in `tempo.yml`):
|
|
|
|
```
|
|
# prefix vs non-prefix hit rate over time
|
|
{ name = "cb.request" } | avg_over_time(span.prefix_hit_rate)
|
|
{ name = "cb.request" } | avg_over_time(span.non_prefix_hit_rate)
|
|
```
|
|
|
|
## Files
|
|
|
|
```
|
|
docker-compose.yml — 4-service stack (collector, tempo, prometheus, grafana)
|
|
otel-collector.yml — OTLP receiver → Tempo + Prometheus fan-out
|
|
tempo.yml — local trace storage + local-blocks TraceQL metrics
|
|
prometheus.yml — scrapes lmcache metrics from collector
|
|
grafana/provisioning/ — auto-provisioned datasources + dashboard
|
|
start-server.sh — launches LMCache server + vLLM with OTLP enabled
|
|
```
|