wehub-resource-sync
33 KiB
LMCache MP Mode Observability Metrics
Overview
The observability system uses an EventBus with pub/sub dispatch and OpenTelemetry for metrics instrumentation.
- Producers (
L1Manager,StorageManager,MPCacheServer) publishEventobjects to the EventBus. - Metrics subscribers (e.g.
L1MetricsSubscriber,L2MetricsSubscriber) subscribe to specific event types and update OTel counters. - Logging subscribers (
MPServerLoggingSubscriber) log events at debug level. - Tracing subscribers (
MPServerTracingSubscriber) create OTel spans from START/END pairs. - Export is via OTLP push to an OTel collector (production) or an in-process
Prometheus
/metricsendpoint (dev/debug fallback).
All metrics use the lmcache_mp. prefix (mp = multiprocess), distinct from the main
engine's lmcache. namespace. On Prometheus, . is converted to _ and counters get
a _total suffix (e.g., lmcache_mp.l1_read with unit="chunks" is exposed as
lmcache_mp_l1_read_chunks_total).
For implementation guidance on adding new events and subscribers, see README.md.
Global Resource Attributes
Every metric (and span) exported by an MP server carries Resource-level attributes built at startup:
| Attribute | CLI flag | Source | Applies to |
|---|---|---|---|
service.instance.id |
--instance-id |
MPServerConfig.instance_id (defaults to a random UUID v4 at startup; projected onto ObservabilityConfig.service_instance_id by run_cache_server so telemetry and coordinator membership share one id) |
All metrics + spans |
Resource attributes are attached to the MeterProvider / TracerProvider
in otel_init.py and therefore appear on every datapoint exported via
OTLP. On Prometheus, SDK resource attributes are typically surfaced via
the target_info series rather than on each time-series.
Per-metric attributes (e.g. cache_salt) remain on the individual
datapoints and are orthogonal to these Resource attributes.
L1 Read Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l1_read |
lmcache_mp_l1_read_chunks_total |
Counter (attr: cache_salt) |
L1_READ_FINISHED |
+len(keys) per cache_salt |
What it answers: How many chunks are being read from L1?
Note:
L1_READ_RESERVEDis published but has no metrics subscriber — key counts are recorded only when the read actually completes.
L1 Write Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l1_write |
lmcache_mp_l1_write_chunks_total |
Counter (attr: cache_salt) |
L1_WRITE_FINISHED |
+len(keys) per cache_salt |
| (same counter) | (same) | Counter (attr: cache_salt) |
L1_WRITE_FINISHED_AND_READ_RESERVED |
+len(keys) per cache_salt |
What it answers: How many chunks are being written to L1?
Note:
L1_WRITE_RESERVEDis published but has no metrics subscriber.L1_WRITE_FINISHED_AND_READ_RESERVED(atomic write-then-read used by prefetch) increments the same write counter.
L1 Eviction Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l1_evicted |
lmcache_mp_l1_evicted_chunks_total |
Counter (attr: cache_salt) |
L1_KEYS_EVICTED |
+len(keys) per cache_salt |
lmcache_mp.l1_eviction_loop_ticks |
lmcache_mp_l1_eviction_loop_ticks_total |
Counter | L1_EVICTION_LOOP_TICK |
+1 per loop iteration |
lmcache_mp.l1_eviction_loop_triggered |
lmcache_mp_l1_eviction_loop_triggered_total |
Counter | L1_EVICTION_LOOP_TICK |
+1 when triggered=True |
lmcache_mp.l1_usage_ratio |
lmcache_mp_l1_usage_ratio |
Observable Gauge | (callback on L1Manager) |
used / total at scrape time |
What it answers: How aggressively is the eviction controller clearing L1? Is the eviction loop alive but staying below the watermark, or actively firing? What is the current L1 fullness?
The two loop counters distinguish "loop is alive" from "eviction fired" — important when debugging short-lived benchmarks (a workload that completes in <1 s never gives the 1Hz polling loop a chance to fire even when usage exceeds the watermark). l1_usage_ratio is registered via :func:register_gauge against L1Manager, so its value reflects current state at scrape time, not a per-tick sample.
L1 Failure Metrics (LM-291 health monitoring)
Tagged counters covering L1 allocation and read failures. The subscriber
groups keys by ObjectKey.model_name to emit a model_name OTel
attribute on every data point, enabling per-model Prometheus slicing
(e.g. lmcache_mp_l1_allocation_failure_total{during="l1_store",model_name="llama-7b"}).
The ticket-specified lmcache_instance_id tag is deferred to a
follow-up: threading it through StorageManager/StoreController would
require a cross-cutting API change out of scope for this PR.
| OTel metric name | Prometheus name | Type | Source event | Calculation | Tags |
|---|---|---|---|---|---|
lmcache_mp.l1_allocation_failure |
lmcache_mp_l1_allocation_failure_chunks_total |
Counter | L1_ALLOCATION_FAILED |
+count per (during, model_name) bucket |
during ∈ {l1_store, l2_prefetch}, model_name |
lmcache_mp.l1_read_failure |
lmcache_mp_l1_read_failure_chunks_total |
Counter | L1_READ_FAILED |
+count per (during, reason, model_name) bucket |
during ∈ {l2_store, l1_retrieve}, reason ∈ {not_found, write_locked}, model_name |
What it answers:
l1_allocation_failure— how often is L1 rejecting writes for lack of memory, split by whether the pressure is user stores or L2 prefetch?l1_read_failure— a post-lookup anomaly counter, not a cache-miss counter. Should stay near zero in healthy operation; any non-zero value indicates a lookup/reserve race or unexpected eviction in MP mode.
Timeout Metrics
Cross-component anomaly counter. Incremented once per LMCacheTimeoutError
constructed (see errors.py and the TIMEOUT_RAISED event in
EVENTS.md), tagged by exception_type so operators can alert on
the timeout rate per class.
| OTel metric name | Prometheus name | Type | Source event | Calculation | Tags |
|---|---|---|---|---|---|
lmcache_mp.timeouts |
lmcache_mp_timeouts_total |
Counter | TIMEOUT_RAISED |
+1 per event |
exception_type |
What it answers: how often are operations timing out, and of which kind? Should stay near zero in healthy operation; a rising rate signals an overloaded or stuck MQ/transfer/adapter path.
L1 Chunk Lifecycle Histograms
Sampled (default 1%) chunk-level lifecycle tracking. Only sampled chunks contribute to histograms; counters above always count all events.
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l1_chunk_lifetime |
lmcache_mp_l1_chunk_lifetime_seconds |
Histogram | L1_KEYS_EVICTED |
eviction_time - alloc_time per sampled chunk |
lmcache_mp.l1_chunk_idle_before_evict |
lmcache_mp_l1_chunk_idle_before_evict_seconds |
Histogram | L1_KEYS_EVICTED |
eviction_time - last_access_time per sampled chunk |
lmcache_mp.l1_chunk_reuse_gap |
lmcache_mp_l1_chunk_reuse_gap_seconds |
Histogram | L1_READ_FINISHED, L1_WRITE_FINISHED, L1_WRITE_FINISHED_AND_READ_RESERVED |
Time gap between consecutive touches of the same chunk |
lmcache_mp.l1_chunk_evict_reuse_gap |
lmcache_mp_l1_chunk_evict_reuse_gap_seconds |
Histogram | L1_KEYS_EVICTED → L1_WRITE_FINISHED |
Time from eviction to next reuse (capped at 300 s) |
lmcache_mp.real_reuse_gap |
lmcache_mp_real_reuse_gap_seconds |
Histogram (tagged cache_salt) |
SM_READ_PREFETCHED_FINISHED, SM_WRITE_FINISHED |
Time gap between a chunk's last access (read or write) and the next read. Captures storage cost. Emitted only on read events. |
lmcache_mp.real_reuse_gap_objects |
lmcache_mp_real_reuse_gap_objects_chunks |
Histogram (tagged cache_salt) |
SM_READ_PREFETCHED_FINISHED, SM_WRITE_FINISHED |
Per-cache_salt access-counter gap between two reads of the same chunk. Counter bumps on every read and write of every chunk; histogram emitted only on read events for sampled chunks. Captures storage volume. |
What it answers: How long do L1 chunks live? How idle are they before eviction? How quickly are evicted chunks reused?
L2 Store Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l2_store_submitted |
lmcache_mp_l2_store_submitted_requests_total |
Counter | L2_STORE_SUBMITTED |
+1 per event |
lmcache_mp.l2_store_submitted_objects |
lmcache_mp_l2_store_submitted_objects_chunks_total |
Counter (attr: cache_salt) |
L2_STORE_SUBMITTED |
+count per cache_salt via key_count_per_salt |
lmcache_mp.l2_store_completed |
lmcache_mp_l2_store_completed_requests_total |
Counter (attr: l2_name) |
L2_STORE_COMPLETED |
+1 per event |
lmcache_mp.l2_store_completed_objects |
lmcache_mp_l2_store_completed_objects_chunks_total |
Counter (attr: cache_salt) |
L2_STORE_COMPLETED |
+count per cache_salt via key_count_per_salt |
What it answers: How many chunks are being pushed to L2? What fraction fail?
L2 Prefetch Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l2_prefetch_lookup |
lmcache_mp_l2_prefetch_lookup_requests_total |
Counter | L2_PREFETCH_LOOKUP_SUBMITTED |
+1 per event |
lmcache_mp.l2_prefetch_lookup_objects |
lmcache_mp_l2_prefetch_lookup_objects_chunks_total |
Counter (attr: cache_salt) |
L2_PREFETCH_LOOKUP_SUBMITTED |
+count per cache_salt via key_count_per_salt |
lmcache_mp.l2_prefetch_hit |
lmcache_mp_l2_prefetch_hit_chunks_total |
Counter | L2_PREFETCH_LOOKUP_COMPLETED |
+prefix_hit_count |
lmcache_mp.l2_prefetch_load_submitted |
lmcache_mp_l2_prefetch_load_submitted_requests_total |
Counter | L2_PREFETCH_LOAD_SUBMITTED |
+adapter_count (per-adapter task count) |
lmcache_mp.l2_prefetch_load_submitted_objects |
lmcache_mp_l2_prefetch_load_submitted_objects_chunks_total |
Counter (attr: cache_salt) |
L2_PREFETCH_LOAD_SUBMITTED |
+count per cache_salt via key_count_per_salt |
lmcache_mp.l2_prefetch_load_completed |
lmcache_mp_l2_prefetch_load_completed_chunks_total |
Counter (attr: cache_salt) |
L2_PREFETCH_LOAD_COMPLETED |
+count per cache_salt via key_count_per_salt |
lmcache_mp.l2_load_completed |
lmcache_mp_l2_load_completed_requests_total |
Counter (attr: l2_name) |
L2_LOAD_TASK_COMPLETED |
+1 per event |
What it answers: How effective is L2 prefetching? What is the L2 hit rate?
Per-backend IOPS. lmcache_mp.l2_store_completed (attr l2_name) counts
completed L1→L2 store tasks; lmcache_mp.l2_load_completed (attr l2_name)
counts completed per-adapter L2→L1 load tasks. Derive per-backend ops/sec on
the dashboard with
rate(lmcache_mp_l2_store_completed_requests_total{l2_name="..."}[1m])
(and the equivalent for loads). No separate *_iops metric is exported — the
raw counter keeps the window choice in the dashboard.
L2 Eviction Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l2_evicted_objects |
lmcache_mp_l2_evicted_chunks_total |
Counter (attr: cache_salt) |
L2_KEYS_EVICTED |
+count per cache_salt via key_count_per_salt |
What it answers: How many chunks are being evicted from L2? Which tenants are losing data?
Lookup Hit-Rate Metrics (L1 + L2 combined)
Token-level counters derived from the MP_LOOKUP_PREFETCH_END event. Their
ratio is the fraction of tokens requested by a lookup that were served from
either L1 or L2. L0 (GPU prefix cache) is intentionally excluded — it is
vLLM-owned and not observable from LMCache.
Both counters carry model_name and cache_salt OTel attributes (captured
at lookup time from IPCCacheServerKey), enabling per-model and per-tenant
slicing of the hit rate. cache_salt can be high-cardinality; drop it at
scrape time with metric_relabel_configs if storage cost matters.
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.lookup_requested |
lmcache_mp_lookup_requested_tokens_total |
Counter (attrs: model_name, cache_salt) |
MP_LOOKUP_PREFETCH_END |
+requested_tokens |
lmcache_mp.lookup_hit |
lmcache_mp_lookup_hit_tokens_total |
Counter (attrs: model_name, cache_salt) |
MP_LOOKUP_PREFETCH_END |
+hit_tokens |
What it answers: What fraction of tokens requested by a lookup were served from cache (L1 or L2)?
# Aggregate hit rate (all models, all salts):
rate(lmcache_mp_lookup_hit_tokens_total[5m])
/ rate(lmcache_mp_lookup_requested_tokens_total[5m])
# Per-model hit rate:
sum(rate(lmcache_mp_lookup_hit_tokens_total[5m])) by (model_name)
/ sum(rate(lmcache_mp_lookup_requested_tokens_total[5m])) by (model_name)
Note: Both counters are driven by the same event, so they always advance together per completed lookup. Early-exit lookups (no GPU context matches, empty
chunk_hashes) contribute0to both, and abandoned lookups (client never pollsquery_prefetch_status) contribute to neither. See L1_L2_HIT_RATE_PLAN.md for the full rationale.
L2 Failure Metrics (LM-291 health monitoring)
| OTel metric name | Prometheus name | Type | Source event | Calculation | Tags |
|---|---|---|---|---|---|
lmcache_mp.l2_prefetch_failure |
lmcache_mp_l2_prefetch_failure_chunks_total |
Counter | L2_PREFETCH_FAILED |
+count per (reason, model_name) bucket |
reason ∈ {l1_oom, not_found}, model_name |
What it answers: For keys L2 reported present at lookup but failed to land in L1: was L1 full (l1_oom), or did the adapter fail to produce the data (not_found)?
Serde failures: a third
reason=serde_failurevalue will be added as an additive, non-breaking extension once the serde PR lands and L2 adapters distinguish deserialization errors from missing objects. No dashboard migration needed when that happens.
TTL lock expiration:
lmcache_mp.l1_ttl_lock_expirefrom the ticket is deferred to a follow-up because the currentTTLLockprimitive (native) has no expiration callback; lazy detection requires a C++/Rust-side change.
L0 (GPU) Block Lifecycle Histograms
Sampled (default 1%) GPU KV cache block lifecycle tracking via shadow monitoring
of MP_VLLM_BLOCK_ALLOCATION and MP_VLLM_END_SESSION events. Eviction is
detected at reallocation time (when a block is assigned different tokens).
All L0 histograms carry instance_id and model_name OTel attributes, enabling
per-instance and per-model Prometheus metric slicing (e.g.
lmcache_mp_l0_block_lifetime_seconds{instance_id="12345",model_name="llama-7b"}).
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l0_block_lifetime |
lmcache_mp_l0_block_lifetime_seconds |
Histogram | MP_VLLM_BLOCK_ALLOCATION (eviction detected) |
eviction_time - alloc_time per sampled block |
lmcache_mp.l0_block_idle_before_evict |
lmcache_mp_l0_block_idle_before_evict_seconds |
Histogram | MP_VLLM_BLOCK_ALLOCATION (eviction detected) |
eviction_time - last_access_time per sampled block |
lmcache_mp.l0_block_reuse_gap |
lmcache_mp_l0_block_reuse_gap_seconds |
Histogram | MP_VLLM_BLOCK_ALLOCATION (cache hit) |
Time gaps between consecutive accesses from access history |
What it answers: How long do GPU blocks live before eviction? How idle are they? How frequently are cached blocks reused? Which instance/model is experiencing the most churn?
L0 ↔ L1 Throughput Histograms
Per-request throughput of GPU↔CPU copies via
L0L1ThroughputSubscriber. Correlates MP_{STORE,RETRIEVE}_START → MP_{STORE,RETRIEVE}_END
pairs by session_id, computes total_bytes / (end_ts - start_ts) in GB/s.
Every request contributes one sample (no sampling).
START/END events fire on the GPU cupy stream (publish_on_stream), so
timestamps reflect true GPU-stream copy time — not Python/lock overhead.
All throughput histograms carry engine_id (vLLM worker instance id),
device (e.g. "cuda:3"), and model_name OTel attributes, enabling
per-worker, per-device, and per-model slicing in Prometheus (e.g.
lmcache_mp_l0_l1_store_throughput_gbs{engine_id="0",device="cuda:3",model_name="meta-llama/Llama-3.1-8B"}).
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l0_l1_store_throughput |
lmcache_mp_l0_l1_store_throughput_GBs |
Histogram | MP_STORE_START → MP_STORE_END |
total_bytes / (end_ts - start_ts) / 1e9 per request |
lmcache_mp.l0_l1_load_throughput |
lmcache_mp_l0_l1_load_throughput_GBs |
Histogram | MP_RETRIEVE_START → MP_RETRIEVE_END |
total_bytes / (end_ts - start_ts) / 1e9 per request |
What it answers: What GPU↔CPU throughput is each vLLM worker actually achieving for KV store/load? Does it match the theoretical PCIe bandwidth? Are some workers or GPUs underperforming?
L1 ↔ L2 Throughput Histograms
Per-task throughput of L1↔L2 transfers via
L2ThroughputSubscriber. The store path correlates L2_STORE_SUBMITTED →
L2_STORE_COMPLETED by (adapter_index, task_id). The load path
correlates the new per-adapter L2_LOAD_TASK_SUBMITTED →
L2_LOAD_TASK_COMPLETED events by (request_id, adapter_index); the
pre-existing request-level L2_PREFETCH_LOAD_* events aggregate across
adapters and cannot attribute throughput to a specific l2_name.
Every task contributes one sample (no sampling).
Unlike the L0↔L1 histograms, these timestamps span submit → complete,
so (end_ts - start_ts) includes adapter queue, network, and disk I/O
time. Treat the value as bytes / end-to-end latency, not raw transfer
rate — useful for comparing adapter types and tracking regressions, not
for validating peak fabric bandwidth.
All throughput histograms carry a single l2_name OTel attribute — the
registered adapter type (e.g. "fs", "nixl_store", "mooncake_store")
— enabling per-adapter-type slicing in Prometheus (e.g.
lmcache_mp_l2_store_throughput_gbs{l2_name="nixl_store"}).
Store-path fast-path accounting. Some adapters skip the write when
a key is already present in the backend, collapsing
(completed_ts - submitted_ts) to near-zero while the submitted
total_bytes count stays unchanged. To avoid inflated throughput
samples, the L2StoreResult returned by pop_completed_store_tasks()
carries bytes_transferred() covering only the bytes actually written.
The L2_STORE_COMPLETED event propagates this value, and the store
throughput subscriber records bytes_transferred / dt; when
bytes_transferred <= 0 (every key fast-pathed) the sample is dropped
entirely. The load path continues to use the submitted total_bytes.
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.l2_store_throughput |
lmcache_mp_l2_store_throughput_GBs |
Histogram | L2_STORE_SUBMITTED → L2_STORE_COMPLETED |
bytes_transferred / (completed_ts - submitted_ts) / 1e9 per task. bytes_transferred is read from the L2_STORE_COMPLETED event (populated from the L2StoreResult returned by pop_completed_store_tasks()); samples where bytes_transferred <= 0 (e.g. duplicate-key fast paths that skip the write) are dropped, so the histogram reflects real work, not submitted-but-skipped bytes. |
lmcache_mp.l2_load_throughput |
lmcache_mp_l2_load_throughput_GBs |
Histogram | L2_LOAD_TASK_SUBMITTED → L2_LOAD_TASK_COMPLETED |
total_bytes / (completed_ts - submitted_ts) / 1e9 per (request, adapter) pair. The load path still uses submitted total_bytes; per-task real-bytes accounting only applies to the store path. |
What it answers: What end-to-end throughput is each L2 adapter delivering? Which backends are keeping up with demand, and which are queue-bound or I/O-bound?
Engine Counters
Worker-scoped counters tied to what the MP server delivers back to each
vLLM worker. Labeled by worker_id — the vLLM worker instance id,
distinct from any scheduler-scoped id used elsewhere.
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_mp.num_chunks_loaded |
lmcache_mp_num_chunks_loaded_total |
Counter (attrs: worker_id, model_name, cache_salt) |
MP_RETRIEVE_END |
+retrieved_count per event |
What it answers: How many LMCache chunks is each vLLM worker loading
from LMCache into its engine? Compare across workers to spot uneven
demand or underserved ranks. Slice by model_name to see per-model
load volume in multi-model deployments, or by cache_salt for per-tenant
attribution (note: cache_salt can be high-cardinality — drop it at
scrape time with metric_relabel_configs if storage cost matters).
EventBus Self-Monitoring
Health metrics for the EventBus itself. The two gauges are registered
inside EventBus.__init__ via register_gauge; the two observable
counters are registered by EventBusSelfMetricsSubscriber. Unlike the
other metrics subscribers, these are not driven by events — they observe
bus state directly via the EventBus accessors and report on every OTel
scrape.
| OTel metric name | Prometheus name | Type | Source | Calculation |
|---|---|---|---|---|
lmcache_mp.event_bus.queue_depth |
lmcache_mp_event_bus_queue_depth |
ObservableGauge | EventBus.queue_depth() |
len(_queue) at scrape time |
lmcache_mp.event_bus.drain_lag_seconds |
lmcache_mp_event_bus_drain_lag_seconds |
ObservableGauge | EventBus.oldest_event_lag_seconds() |
time.time() - oldest.timestamp, or 0.0 when empty |
lmcache_mp.event_bus.dropped_events_total |
lmcache_mp_event_bus_dropped_events_total |
ObservableCounter | EventBus.dropped_events_count() |
cumulative _discard_count |
lmcache_mp.event_bus.subscriber_exceptions |
lmcache_mp_event_bus_subscriber_exceptions_total |
ObservableCounter (attr: subscriber_name) |
EventBus.subscriber_exception_counts() |
cumulative count per subscriber, incremented when _drain_all catches a callback exception |
What it answers: Is the EventBus keeping up with publishers? Is anything being dropped? Are any subscriber callbacks raising?
subscriber_name is derived from the failing callback: bound methods report their owning class (e.g. L1MetricsSubscriber); free functions report __qualname__.
MPCacheServer Observable Gauges
These metrics are registered directly via register_gauge (pull-based OTel
observable gauges) rather than through the EventBus, because they represent
point-in-time state snapshots that do not correspond to discrete events.
| OTel metric name | Prometheus name | Type | Source | Calculation |
|---|---|---|---|---|
lmcache_mp.active_prefetch_jobs |
lmcache_mp_active_prefetch_jobs |
ObservableGauge | MPCacheServer._prefetch_jobs |
len(_prefetch_jobs) at scrape time |
What it answers: How many prefetch jobs are currently in-flight? A sustained high value may indicate slow L2 backends or client-side polling delays.
L1 / L2 State Metrics
Live state of the L1 memory pool, the per-adapter L2 byte usage, and the in-flight L2 store / prefetch-load queues. These metrics are useful for capacity planning, sizing L1, watching L2 fullness, and spotting backpressure on individual L2 adapters.
All five metrics are OTel ObservableGauge instruments registered via the
shared register_gauge helper. At scrape time, OTel invokes the
registered callback, which iterates the controller's live state and
returns one observation per adapter (for l2_usage_bytes, one per
configured adapter; for the in-flight gauges, only adapters with work).
Adapters with no in-flight work emit no datapoint for the three
in-flight gauges.
lmcache_mp.l2_usage_bytes carries a single l2_name attribute — the
adapter's registered type name (e.g. "fs", "mock",
"nixl_store"). The three in-flight metrics carry two attributes
that disambiguate adapters even when more than one is registered with
the same backend type — same shape as the existing
lmcache_mp.l2_store_completed counter:
l2_name— the registered adapter type (e.g."fs","mock","nixl_store").adapter_index— position in theStoreController/PrefetchControlleradapter list. Distinguishes two adapters of the same type (e.g.fs[0]andfs[1]).
| OTel metric name | Prometheus name | Type | Source of truth | Calculation |
|---|---|---|---|---|
lmcache_mp.l1_memory_usage_bytes |
lmcache_mp_l1_memory_usage_bytes |
ObservableGauge | L1Manager.get_memory_usage() |
Bytes currently held in L1 at scrape time |
lmcache_mp.l2_usage_bytes |
lmcache_mp_l2_usage_bytes |
ObservableGauge (attr: l2_name) |
StorageManager.get_l2_usages() (calls L2AdapterInterface.get_usage().total_bytes_used) |
Per-adapter bytes currently held in L2 at scrape time; one observation per configured adapter. Adapters whose get_usage() raises are skipped silently. |
lmcache_mp.num_inflight_l2_stores |
lmcache_mp_num_inflight_l2_stores |
ObservableGauge (attrs: l2_name, adapter_index) |
StoreController.get_inflight_count_by_adapter() |
Snapshot of in-flight L2 store tasks grouped by adapter |
lmcache_mp.num_inflight_l2_loads |
lmcache_mp_num_inflight_l2_loads |
ObservableGauge (attrs: l2_name, adapter_index) |
PrefetchController.get_inflight_load_state_by_adapter() |
Per-adapter count from the same snapshot |
lmcache_mp.inflight_load_memory_usage_bytes |
lmcache_mp_inflight_load_memory_usage_bytes |
ObservableGauge (attrs: l2_name, adapter_index) |
PrefetchController.get_inflight_load_state_by_adapter() |
Per-adapter reserved bytes from the same snapshot |
What l1_memory_usage_bytes answers: How full is the L1 cache? Helps
size L1 against working set and detect leaks (steadily climbing without
plateauing).
What l2_usage_bytes answers: How full is each L2 backend? Lets
operators query how much each L2 tier currently holds, decide whether
an adapter needs eviction or purge, and spot per-backend asymmetries
when more than one L2 is configured. Parallel to l1_memory_usage_bytes
on the L2 tier.
What num_inflight_l2_stores answers: Are L2 stores piling up on a
particular adapter? Sustained non-zero values indicate the adapter cannot
keep up with the L1 → L2 write rate.
What num_inflight_l2_loads answers: Are L2 → L1 prefetch loads
backing up? Pair with num_inflight_l2_stores to see whether read or
write traffic dominates a given backend.
What inflight_load_memory_usage_bytes answers: How much L1 capacity
is currently reserved but not yet filled by in-flight prefetches? Rising
in-flight bytes alongside rising l1_memory_usage_bytes is a signal that
prefetch reservations are crowding out cacheable data.
Bytes attribution. A single prefetch request may load from multiple adapters. The byte count is split per-adapter via the request's
load_planbitmap × per-keyMemoryObj.size(precomputed at submit time and stored onInFlightPrefetchRequest.load_bytes_by_adapter) so each in-flight byte is attributed to exactly one(l2_name, adapter_index)pair — sums across adapters are not double-counted.
Singleton dispatch. L1Manager / StoreController / PrefetchController are singletons in MP mode. Each controller registers its gauge once (guarded by a class-level
_gauge_registeredflag) and the callback dispatches via a class-level_gauge_targetso the most recently constructed instance owns the reported values. This is invisible in production (one instance per process); it matters in tests that create multiple controllers.
Thread safety. Callbacks run on the OTel reader thread and read state mutated by the controller's background loop thread. Snapshots use
dict.copy(), which is implemented in C and atomic under the CPython GIL — concurrent mutation cannot crash the snapshot, though it may briefly see a state that is one mutation stale. Acceptable for a 10-second scrape cadence.
Cache Blending (CB) Metrics
Metrics for Cache Blending operations use the lmcache_blend. prefix (distinct from the
MP mode lmcache_mp. namespace). On Prometheus, . becomes _ and counters get
_total suffix (e.g., lmcache_blend_lookup_requests_total).
CB Lookup Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_blend.lookup_requests |
lmcache_blend_lookup_requests_total |
Counter | CB_LOOKUP_START |
+1 per event |
lmcache_blend.lookup_requested_tokens |
lmcache_blend_lookup_requested_tokens_total |
Counter | CB_LOOKUP_END |
+requested_tokens |
lmcache_blend.lookup_hit_tokens |
lmcache_blend_lookup_hit_tokens_total |
Counter | CB_LOOKUP_END |
+hit_tokens |
lmcache_blend.lookup_fingerprint_hits |
lmcache_blend_lookup_fingerprint_hits_total |
Counter | CB_LOOKUP_END |
+fingerprint_hits |
lmcache_blend.lookup_storage_hits |
lmcache_blend_lookup_storage_hits_total |
Counter | CB_LOOKUP_END |
+storage_hits |
lmcache_blend.lookup_stale_chunks |
lmcache_blend_lookup_stale_chunks_total |
Counter | CB_LOOKUP_END |
+stale_chunks |
lmcache_blend.lookup_no_gpu_context_errors |
lmcache_blend_lookup_no_gpu_context_errors_total |
Counter | CB_LOOKUP_END |
+1 when no_gpu_context=True |
What it answers: How often does the CB server receive lookup requests? What fraction of requested tokens are served by blend (token-level hit rate)? What fraction hit the fingerprint table? What fraction are confirmed in storage? How many stale evictions occur?
Blend token-level hit rate (numerator and denominator co-emit on the same CB_LOOKUP_END event so the ratio is meaningful even under partial-failure paths):
rate(lmcache_blend_lookup_hit_tokens_total[5m])
/ rate(lmcache_blend_lookup_requested_tokens_total[5m])
CB Retrieve Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_blend.retrieve_requests |
lmcache_blend_retrieve_requests_total |
Counter | CB_RETRIEVE_START |
+1 per event |
lmcache_blend.retrieve_chunks |
lmcache_blend_retrieve_chunks_total |
Counter | CB_RETRIEVE_START |
+num_chunks |
lmcache_blend.retrieve_failures |
lmcache_blend_retrieve_failures_total |
Counter | CB_RETRIEVE_END |
+1 when success=False |
What it answers: How often is CB retrieval invoked? How many chunks are retrieved per call? What is the failure rate?
CB Store Pre-computed Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_blend.store_pre_computed_requests |
lmcache_blend_store_pre_computed_requests_total |
Counter | CB_STORE_PRE_COMPUTED_START |
+1 per event |
lmcache_blend.store_pre_computed_chunks |
lmcache_blend_store_pre_computed_chunks_total |
Counter | CB_STORE_PRE_COMPUTED_END |
+stored_chunks |
lmcache_blend.store_pre_computed_failures |
lmcache_blend_store_pre_computed_failures_total |
Counter | CB_STORE_PRE_COMPUTED_END |
+1 when success=False |
What it answers: How often is pre-computed CB storage invoked? How many chunks are written? What is the failure rate?
CB Store Final Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_blend.store_final_requests |
lmcache_blend_store_final_requests_total |
Counter | CB_STORE_FINAL_START |
+1 per event |
lmcache_blend.store_final_chunks |
lmcache_blend_store_final_chunks_total |
Counter | CB_STORE_FINAL_END |
+stored_chunks |
lmcache_blend.store_final_failures |
lmcache_blend_store_final_failures_total |
Counter | CB_STORE_FINAL_END |
+1 when success=False |
What it answers: How often is final CB storage invoked? How many chunks are committed? What is the failure rate?
CB Fingerprint Table Metrics
| OTel metric name | Prometheus name | Type | Source event | Calculation |
|---|---|---|---|---|
lmcache_blend.fingerprints_registered |
lmcache_blend_fingerprints_registered_total |
Counter | CB_FINGERPRINTS_REGISTERED |
+num_chunks |
lmcache_blend.chunks_evicted |
lmcache_blend_chunks_evicted_total |
Counter | CB_CHUNKS_EVICTED |
+num_chunks |
What it answers: How many chunks are indexed into the fingerprint table? How many stale entries are evicted?
For derivations of L1-only / L2-only / blend hit rates from these counters, and a "what to send when reporting" checklist, see DEBUG.md.
For event metadata contracts (what keys each EventType carries), see
EVENTS.md.