LMCache / vLLM
  └─ OTLP gRPC → OTel Collector (:4320)
                   ├─ traces  → Tempo (:3200)
                   └─ metrics → Prometheus (:9091)
                                  └─ Grafana (:3000)

Step 1 — Start the observability stack

cd examples/observability
docker compose up -d

Step 2 — Start LMCache + vLLM

MODEL=/your/model/path bash start-server.sh

Step 3 — Send requests to populate traces

# Run a short long-doc-qa benchmark: first query is a miss, subsequent
# queries against the same document are cache hits.
lmcache bench engine \
  --engine-url http://localhost:8100 \
  --workload long-doc-qa \
  --kv-cache-volume 1 \
  --ldqa-query-per-document 10

Step 4 — Visualize in Grafana

Open http://localhost:3000 → Explore → datasource Tempo.

# All request root spans
{ name = "request" }

# Filter to a specific session
{ name = "request" && span.session_id = "<request_id>" }

# Only cache-hit requests (had a retrieve)
{ name = "request" } >> { name = "mp.retrieve" }

# Requests with less than 50 % cache hit rate
{ name = "request" && span.hit_rate < 0.5 }

# Full cache hits only
{ name = "request" && span.hit_rate = 1.0 }

# Complete misses (lookup ran but nothing was cached)
{ name = "request" && span.requested_tokens > 0 && span.hit_tokens = 0 }

Click any trace to open the waterfall. Each root request span carries three per-request cache hit rate attributes:

Attribute	Type	Description
`hit_tokens`	int	tokens served from L1+L2 cache
`requested_tokens`	int	total chunk-aligned tokens submitted for lookup
`hit_rate`	float	`hit_tokens / requested_tokens` (0.0 on a total miss)

request  [══════════════════════════════════════]  hit_rate=0.75
  mp.lookup_prefetch  [════]
  mp.retrieve               [════════]
  mp.store                            [══════]

Store-only requests (no lookup phase) do not carry these attributes.

The pre-provisioned LMCache dashboard under Dashboards shows cache hit rate, StorageManager read/write rates, and the live trace panel. The collapsed CacheBlend row adds blend-server panels (see below).

CacheBlend (blend server) traces

When LMCache runs the blend engine (lmcache server --engine-type blend), CacheBlend V3 emits its own span tree to Tempo alongside the standard spans. Expand the collapsed CacheBlend row on the dashboard, or query Tempo:

# All CacheBlend request traces
{ name = "cb.request" }

# Requests that actually blended non-prefix (shifted) KV
{ name = "cb.request" && span.non_prefix_hit_tokens > 0 }

# The token-scatter GPU step
{ name = "cb.scatter" }

Click a cb.request row to open the waterfall:

cb.request
  cb.lookup                (attr prefix_chunks; prefix timing is in mp.lookup_prefetch)
    cb.fingerprint_match   match probe hashes vs stored fingerprints
    cb.sparse_prefetch     non-prefix (shifted) chunks, sparse L2->L1
                           (emitted only on an actual L2 load; carries l2_keys)
  cb.retrieve
    cb.scatter             L1 -> paged KV per-token slot-scatter + re-RoPE
  cb.store_pre_computed
  cb.store_final

The root cb.request span carries the V3 hit-rate breakdown (hit_rate = prefix + non-prefix):

Attribute	Type	Description
`prefix_hit_tokens`	int	tokens reused from the prefix (L1+L2)
`non_prefix_hit_tokens`	int	tokens reused from sparse non-prefix chunks
`hit_tokens`	int	`prefix_hit_tokens + non_prefix_hit_tokens`
`requested_tokens`	int	total chunk-aligned tokens submitted
`hit_rate`	float	`hit_tokens / requested_tokens`
`prefix_hit_rate`	float	`prefix_hit_tokens / requested_tokens`
`non_prefix_hit_rate`	float	`non_prefix_hit_tokens / requested_tokens` (sums to `hit_rate`)

The CacheBlend Hit Rate & Chunks panel overlays the overall token hit rate (Prometheus) with the per-request prefix/non-prefix breakdown via TraceQL metrics, served by Tempo's local-blocks metrics generator (enabled in tempo.yml):

# prefix vs non-prefix hit rate over time
{ name = "cb.request" } | avg_over_time(span.prefix_hit_rate)
{ name = "cb.request" } | avg_over_time(span.non_prefix_hit_rate)

Files

docker-compose.yml          — 4-service stack (collector, tempo, prometheus, grafana)
otel-collector.yml          — OTLP receiver → Tempo + Prometheus fan-out
tempo.yml                   — local trace storage + local-blocks TraceQL metrics
prometheus.yml              — scrapes lmcache metrics from collector
grafana/provisioning/       — auto-provisioned datasources + dashboard
start-server.sh             — launches LMCache server + vLLM with OTLP enabled