Metrics ======= Metrics are collected via OpenTelemetry and made available to Prometheus in one of two ways, depending on whether ``--otlp-endpoint`` is set: - **Pull mode (default, no collector).** When ``--otlp-endpoint`` is *not* set, the server publishes a Prometheus ``/metrics`` endpoint that Prometheus scrapes directly. .. important:: For ``lmcache server``, ``/metrics`` is served by the **HTTP frontend** on ``--http-port`` (default **8080**), e.g. ``http://:8080/metrics`` — **not** on ``--prometheus-port``. ``--prometheus-port`` is *ignored* by ``lmcache server``: the standalone Prometheus HTTP server is disabled because the HTTP frontend already serves ``/metrics``. The frontend-less entrypoints (``python -m lmcache.v1.multiprocess.server`` and ``lmcache trace replay``) have no HTTP frontend, so *they* serve ``/metrics`` on ``--prometheus-port`` (default 9090). See :ref:`mp-obs-metrics-endpoint` for the full breakdown. Either way, ``/metrics`` is empty until the first store/retrieve — drive some traffic before you go looking. - **Push mode (OTLP).** When ``--otlp-endpoint`` is set, metrics are pushed to an OpenTelemetry Collector, which re-exposes them for Prometheus to scrape. See :doc:`index` for the bundled Collector + Prometheus + Grafana stack. All metrics use the ``lmcache_mp.`` prefix (multiprocess). On Prometheus, dots are converted to underscores and counters get a ``_total`` suffix (e.g. ``lmcache_mp_l1_read_chunks_total``); histograms gain a unit suffix plus ``_sum`` / ``_count`` / ``_bucket`` (e.g. ``lmcache_mp_l2_store_throughput_GB_per_second_sum``). .. _mp-observability-resource: Global Resource Attributes ~~~~~~~~~~~~~~~~~~~~~~~~~~ Every metric and span exported by an MP server carries Resource-level attributes built at startup. These identify the process producing the telemetry and are orthogonal to per-metric attributes such as ``cache_salt``. .. list-table:: :header-rows: 1 :widths: 30 25 45 * - Attribute - CLI flag / config - Default when unset * - ``service.instance.id`` - ``--instance-id`` / ``MPServerConfig.instance_id`` - Random UUID v4 minted at startup. Resource attributes attach to the ``MeterProvider`` / ``TracerProvider`` and propagate to every exported datapoint via OTLP. On Prometheus, SDK resource attributes surface on the ``target_info`` series rather than on each time-series — this is standard OTel behavior. L1 Metrics ~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l1_read`` - Counter (attr: ``cache_salt``) - Number of chunks read from L1, grouped by tenant. * - ``lmcache_mp.l1_write`` - Counter (attr: ``cache_salt``) - Number of chunks written to L1, grouped by tenant. * - ``lmcache_mp.l1_evicted`` - Counter (attr: ``cache_salt``) - Number of chunks evicted by the EvictionController, grouped by tenant. * - ``lmcache_mp.l1_eviction_loop_ticks`` - Counter - L1 eviction-loop iterations (every cycle, regardless of whether the watermark was crossed). Driven by ``L1_EVICTION_LOOP_TICK``. * - ``lmcache_mp.l1_eviction_loop_triggered`` - Counter - L1 eviction-loop iterations where ``usage >= watermark`` and the eviction policy actually ran. The two counters distinguish "loop is alive" from "eviction fired" — important when debugging short-lived benchmarks that complete faster than the 1 Hz polling cycle. L1 Chunk Lifecycle Histograms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sampled (default 1%) chunk-level lifecycle tracking via ``L1LifecycleSubscriber``. Only sampled chunks contribute to histograms; counters above always count all events. Sampling is deterministic (hash-based), so the same key always gets the same decision with zero memory overhead. .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l1_chunk_lifetime`` - Histogram - Time from allocation to eviction per sampled chunk. * - ``lmcache_mp.l1_chunk_idle_before_evict`` - Histogram - Time from last access to eviction per sampled chunk. * - ``lmcache_mp.l1_chunk_reuse_gap`` - Histogram - Time gap between consecutive touches (read or write) of the same chunk. * - ``lmcache_mp.l1_chunk_evict_reuse_gap`` - Histogram - Time from eviction to next reuse (capped at 300 s). StorageManager Real-Reuse Metrics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Workload-level reuse histograms emitted by ``SMLifecycleSubscriber``, driven by caller-facing StorageManager events (``SM_READ_PREFETCHED_FINISHED``, ``SM_WRITE_FINISHED``). Internal read-lock releases by the store/prefetch controllers are excluded so the signal reflects user-driven access only. Both histograms are tagged with ``cache_salt`` for per-tenant isolation. The per-salt access counter advances on every read and write of every chunk (regardless of sampling) so the chunks-gap reflects true storage volume; the histogram itself records gaps only for chunks that pass the (deterministic, hash-based) sampling gate. .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.real_reuse_gap`` - Histogram (tag: ``cache_salt``) - Time gap between a chunk's last access (read or write) and its next read. Captures storage cost — how long a stored chunk sat between accesses. Emitted only on read events. * - ``lmcache_mp.real_reuse_gap_objects`` - Histogram (tag: ``cache_salt``) - Per-``cache_salt`` access-counter gap between two reads of the same chunk. Captures storage volume — how many chunk-accesses occurred while this chunk waited for its next read. Emitted on read events for sampled chunks. L2 Metrics ~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l2_store_submitted`` - Counter - Number of L2 store requests submitted. * - ``lmcache_mp.l2_store_submitted_objects`` - Counter (attr: ``cache_salt``) - Number of chunks submitted for L2 store, grouped by tenant. * - ``lmcache_mp.l2_store_completed`` - Counter (attr: ``l2_name``) - Number of L2 store requests completed, labeled by adapter type. * - ``lmcache_mp.l2_store_completed_objects`` - Counter (attr: ``cache_salt``) - Number of chunks successfully stored to L2, grouped by tenant. * - ``lmcache_mp.l2_prefetch_lookup`` - Counter - Number of L2 prefetch lookup requests. * - ``lmcache_mp.l2_prefetch_lookup_objects`` - Counter (attr: ``cache_salt``) - Number of chunks submitted for L2 prefetch lookup, grouped by tenant. * - ``lmcache_mp.l2_prefetch_hit`` - Counter - Number of prefix chunks found in L2 lookup. * - ``lmcache_mp.l2_prefetch_load_submitted`` - Counter - Number of L2 prefetch load requests submitted. * - ``lmcache_mp.l2_prefetch_load_submitted_objects`` - Counter (attr: ``cache_salt``) - Number of chunks submitted for L2 load, grouped by tenant. * - ``lmcache_mp.l2_prefetch_load_completed`` - Counter (attr: ``cache_salt``) - Number of chunks successfully loaded from L2, grouped by tenant. * - ``lmcache_mp.l2_load_completed`` - Counter (attr: ``l2_name``) - Number of per-adapter L2 load requests completed, labeled by adapter type. * - ``lmcache_mp.l2_evicted_objects`` - Counter (attr: ``cache_salt``) - Number of chunks evicted from L2, grouped by tenant. The ``l2_name``-labeled counters (``l2_store_completed`` and ``l2_load_completed``) exist so dashboards can compute per-backend IOPS on demand via ``rate(lmcache_mp_l2_store_completed_requests_total{l2_name="..."}[1m])`` (and the equivalent for loads). No separate ``*_iops`` metric is exported; keeping the raw counter lets dashboard users pick their own window. Failure & Health Counters ~~~~~~~~~~~~~~~~~~~~~~~~~ Health-monitoring counters emitted on the dedicated ``lmcache_mp.health`` OTel meter. Driven by the ``L1FailureMetricsSubscriber`` and ``L2FailureMetricsSubscriber``, which are registered automatically when metrics are enabled. All three counters carry ``model_name`` (extracted from each ``ObjectKey``) so operators can slice per-model on the Prometheus ``/metrics`` endpoint. .. list-table:: :header-rows: 1 :widths: 35 15 50 * - Metric - Type - Description * - ``lmcache_mp.l1_allocation_failure`` - Counter - L1 memory allocation failures (OOM) during ``reserve_write``. Tagged by ``during`` ∈ {``l1_store``, ``l2_prefetch``} to distinguish user-initiated stores from prefetch-triggered allocations, plus ``model_name``. * - ``lmcache_mp.l1_read_failure`` - Counter - L1 ``reserve_read`` failures. Tagged by ``during`` ∈ {``l2_store``, ``l1_retrieve``}, ``reason`` ∈ {``not_found``, ``write_locked``}, plus ``model_name``. **Post-lookup anomaly counter**, not a cache-miss counter — in MP mode ``reserve_read`` is only called after a successful lookup, so any non-zero value indicates a lookup/reserve race or unexpected eviction and should stay near zero in healthy operation. * - ``lmcache_mp.l2_prefetch_failure`` - Counter - Chunks that L2 reported present at lookup but failed to land in L1. Tagged by ``reason`` ∈ {``l1_oom``, ``not_found``} plus ``model_name``. ``l1_oom`` means L1 had no room to receive the prefetched object; ``not_found`` means the adapter returned no data despite a positive lookup (e.g. concurrent delete). A ``reason=serde_failure`` value will be added to ``l2_prefetch_failure`` as an additive, non-breaking extension once L2 adapters distinguish deserialization errors from missing objects — no dashboard migration needed when that lands. For the full design rationale (including which event types drive each counter and why ``lmcache_instance_id`` is deferred), see ``docs/design/v1/mp_observability/METRICS.md`` in the source tree. Lookup Hit-Rate Metrics ~~~~~~~~~~~~~~~~~~~~~~~ Token-level counters whose ratio gives the fraction of tokens requested by a lookup that were served from either L1 or L2. L0 (GPU prefix cache) is intentionally excluded — it is vLLM-owned and not observable from LMCache. .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.lookup_requested`` - Counter (attrs: ``model_name``, ``cache_salt``) - Total tokens submitted for lookup (denominator of the L1+L2 token-level hit rate). Only chunk-aligned tokens are counted. * - ``lmcache_mp.lookup_hit`` - Counter (attrs: ``model_name``, ``cache_salt``) - Total tokens found in L1 or L2 during lookup (numerator of the L1+L2 token-level hit rate). Counts the contiguous prefix hit only. Both counters are driven by the same event (``MP_LOOKUP_PREFETCH_END``), so they always advance together per completed lookup. Early-exit lookups contribute ``0`` to both, and abandoned lookups contribute to neither. The ``model_name`` and ``cache_salt`` attributes are captured at lookup time from ``IPCCacheServerKey`` so dashboards can compute per-model or per-tenant hit rate. ``cache_salt`` can be high-cardinality (one entry per tenant or isolation domain); drop it at scrape time with ``metric_relabel_configs`` if storage cost matters. **PromQL for hit rate:** .. code-block:: promql # Aggregate (all models, all salts): rate(lmcache_mp_lookup_hit_tokens_total[5m]) / rate(lmcache_mp_lookup_requested_tokens_total[5m]) # Per-model: sum(rate(lmcache_mp_lookup_hit_tokens_total[5m])) by (model_name) / sum(rate(lmcache_mp_lookup_requested_tokens_total[5m])) by (model_name) L0 (GPU) Block Lifecycle Histograms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sampled (default 1%) GPU KV cache block lifecycle tracking via ``L0LifecycleSubscriber``. Eviction is detected at reallocation time (when a block is assigned different tokens). Sampling uses random selection with a ``_skipped`` set (bounded by the finite number of physical GPU blocks). All L0 histograms are emitted with ``instance_id`` and ``model_name`` OTel attributes, enabling per-instance and per-model metric slicing in Prometheus (e.g. ``lmcache_mp_l0_block_lifetime_seconds{instance_id="12345",model_name="llama-7b"}``). .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l0_block_lifetime`` - Histogram - Time from allocation to eviction per sampled GPU block. * - ``lmcache_mp.l0_block_idle_before_evict`` - Histogram - Time from last access to eviction per sampled GPU block. * - ``lmcache_mp.l0_block_reuse_gap`` - Histogram - Time gaps between consecutive accesses of the same GPU block. L0 ↔ L1 Throughput Histograms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Per-request throughput of GPU↔CPU copies via ``L0L1ThroughputSubscriber``. Every store/retrieve request contributes one sample to the appropriate histogram: ``total_bytes / (end_ts - start_ts)`` in GB/s. Timestamps come from ``MP_{STORE,RETRIEVE}_{START,END}`` events published on the GPU cupy stream, so they reflect true GPU-stream copy time — not Python/lock overhead. All throughput histograms are emitted with ``engine_id`` (vLLM worker instance id), ``device`` (e.g. ``"cuda:3"``), and ``model_name`` OTel attributes, enabling per-worker, per-device, and per-model slicing in Prometheus (e.g. ``lmcache_mp_l0_l1_store_throughput_GB_per_second{engine_id="0",device="cuda:3",model_name="meta-llama/Llama-3.1-8B"}``). .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l0_l1_store_throughput`` - Histogram - GPU→CPU (L0→L1) store throughput in GB/s per request. * - ``lmcache_mp.l0_l1_load_throughput`` - Histogram - CPU→GPU (L1→L0) load throughput in GB/s per request. L1 ↔ L2 Throughput Histograms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Per-request throughput of L1↔L2 transfers via ``L2ThroughputSubscriber``. The store path correlates ``L2_STORE_SUBMITTED`` → ``L2_STORE_COMPLETED`` by ``(adapter_index, task_id)``. The load path correlates the per-adapter ``L2_LOAD_TASK_SUBMITTED`` → ``L2_LOAD_TASK_COMPLETED`` events by ``(request_id, adapter_index)``; the request-level ``L2_PREFETCH_LOAD_*`` events used by the chunk-count counters aggregate across adapters and cannot be attributed to a specific ``l2_name``. Timestamps span **submit → complete**, so the duration includes adapter queue, network, and disk I/O — the value is *bytes / end-to-end latency*, not raw transfer rate. Use these histograms to compare adapter types and catch regressions; use the L0↔L1 histograms when you need pure copy-time throughput. All L1↔L2 throughput histograms carry a single ``l2_name`` OTel attribute — the registered adapter type (e.g. ``"fs"``, ``"nixl_store"``, ``"mooncake_store"``) — enabling per-backend slicing in Prometheus (e.g. ``lmcache_mp_l2_store_throughput_GB_per_second{l2_name="nixl_store"}``). .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.l2_store_throughput`` - Histogram - L1→L2 store throughput in GB/s per request. * - ``lmcache_mp.l2_load_throughput`` - Histogram - L2→L1 load throughput in GB/s per (request, adapter) pair. **PromQL for average throughput (GB/s), per backend:** .. code-block:: promql # L1 -> L2 store throughput, averaged over the last minute, per l2_name: sum by (l2_name) (rate(lmcache_mp_l2_store_throughput_GB_per_second_sum[1m])) / sum by (l2_name) (rate(lmcache_mp_l2_store_throughput_GB_per_second_count[1m])) # L2 -> L1 load throughput (same shape): sum by (l2_name) (rate(lmcache_mp_l2_load_throughput_GB_per_second_sum[1m])) / sum by (l2_name) (rate(lmcache_mp_l2_load_throughput_GB_per_second_count[1m])) .. note:: ``l2_store_throughput`` populates whenever chunks are written to L2. ``l2_load_throughput`` only populates when chunks are read **from L2 into L1** — i.e. on a prefetch load after the entry has aged out of L1. If your working set fits entirely in L1 (common with small models or a large ``--l1-size-gb``), lookups are served from L1 and the load histogram stays empty even though store throughput is non-zero. Drive enough distinct data to force L1 eviction, or restart the server between store and load passes, to exercise the L2 load path. Engine Counters ~~~~~~~~~~~~~~~ Worker-scoped counters tied to what the MP server delivers back to each vLLM worker via ``retrieve()``. Labeled by ``worker_id`` (the vLLM worker instance id) — distinct from any scheduler-scoped id that may appear on other metrics. .. list-table:: :header-rows: 1 :widths: 40 25 35 * - Metric - Type - Description * - ``lmcache_mp.num_chunks_loaded`` - Counter (attrs: ``worker_id``, ``model_name``, ``cache_salt``) - Total number of LMCache chunks loaded into the engine, summed over all ``retrieve()`` completions. Sliceable per worker, per model, and per tenant / isolation domain (``cache_salt``). ``cache_salt`` may be high-cardinality; drop it at scrape time with ``metric_relabel_configs`` if storage cost matters. Observable Gauges ~~~~~~~~~~~~~~~~~ Point-in-time state snapshots registered via ``register_gauge`` (pull-based OTel observable gauges). The three in-flight metrics carry two attributes that distinguish adapters even when more than one is registered with the same backend type — same shape as ``lmcache_mp.l2_store_completed``: - ``l2_name`` — the registered adapter type (e.g. ``"fs"``, ``"nixl_store"``, ``"mooncake_store"``). - ``adapter_index`` — position in the controller's adapter list. Adapters with no in-flight work emit no datapoint for that scrape. .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.active_prefetch_jobs`` - ObservableGauge - Number of prefetch jobs currently in-flight. A sustained high value may indicate slow L2 backends or polling delays. * - ``lmcache_mp.l1_memory_usage_bytes`` - ObservableGauge - Bytes currently held in L1. Rising without plateauing typically indicates a leak; saturating at the configured ``--l1-size-gb`` indicates working set exceeds capacity. * - ``lmcache_mp.l1_usage_ratio`` - ObservableGauge - L1 used/total ratio (``0.0``–``1.0``), sampled at scrape time from ``L1Manager.get_memory_usage()``. Returns ``0.0`` when the gauge target is not yet wired up or ``total_bytes`` is zero, so the callback never raises during a scrape. Compare against the eviction watermark (default ``0.8``) to read whether the eviction loop is below or above its trigger threshold. * - ``lmcache_mp.l2_usage_bytes`` - ObservableGauge (attr: ``l2_name``) - Bytes currently held in each L2 adapter, sampled at scrape time from ``adapter.get_usage()``. One observation per configured adapter, tagged by ``l2_name`` (the adapter type, e.g. ``"fs"``, ``"nixl_store"``, ``"mooncake_store"``). Parallel to ``l1_memory_usage_bytes`` for the L2 tier — use it to see how much each L2 backend currently holds. Adapters whose ``get_usage()`` raises are skipped silently rather than poisoning the observation, so a missing datapoint for one ``l2_name`` can mean either "not configured" or "adapter errored on this scrape" — cross-check with the L2 store/load counters. * - ``lmcache_mp.num_inflight_l2_stores`` - ObservableGauge (attrs: ``l2_name``, ``adapter_index``) - L2 store tasks currently executing, per adapter. Sustained non-zero values indicate the adapter cannot keep up with the L1 → L2 write rate. * - ``lmcache_mp.num_inflight_l2_loads`` - ObservableGauge (attrs: ``l2_name``, ``adapter_index``) - L2 → L1 prefetch load tasks currently executing, per adapter. Pair with ``num_inflight_l2_stores`` to see whether read or write traffic dominates a given backend. * - ``lmcache_mp.inflight_load_memory_usage_bytes`` - ObservableGauge (attrs: ``l2_name``, ``adapter_index``) - L1 bytes reserved by in-flight L2 → L1 prefetch loads, per adapter. Rising in-flight bytes alongside rising ``l1_memory_usage_bytes`` is a signal that prefetch reservations are crowding out cacheable data. Per-adapter byte attribution follows each request's ``load_plan`` bitmap, so summing across adapters never double-counts. EventBus Self-Monitoring ~~~~~~~~~~~~~~~~~~~~~~~~ Health metrics for the EventBus itself, registered by ``EventBusSelfMetricsSubscriber`` on the ``lmcache.event_bus`` OTel meter. These metrics observe bus state directly via the ``EventBus`` accessors and report on every OTel scrape — they are not driven by events, so dropping or failing subscribers cannot silence them. Use them to answer: is the EventBus keeping up with publishers, is anything being dropped, and are any subscriber callbacks raising? A non-zero ``dropped_events_total`` or a sustained non-zero ``drain_lag_seconds`` indicates the bus is at ``--event-bus-queue-size`` and tail-dropping; raise that flag or investigate slow subscribers. .. list-table:: :header-rows: 1 :widths: 40 15 45 * - Metric - Type - Description * - ``lmcache_mp.event_bus.queue_depth`` - ObservableGauge - Events currently queued in the EventBus (``len(_queue)`` at scrape time). * - ``lmcache_mp.event_bus.drain_lag_seconds`` - ObservableGauge - Seconds since the oldest queued event was published; ``0.0`` when empty. Rising values mean the drain thread is falling behind. * - ``lmcache_mp.event_bus.dropped_events_total`` - ObservableCounter - Cumulative events dropped because the EventBus queue was at ``--event-bus-queue-size``. * - ``lmcache_mp.event_bus.subscriber_exceptions`` - ObservableCounter (attr: ``subscriber_name``) - Cumulative exceptions raised by subscriber callbacks during EventBus dispatch, tagged by ``subscriber_name`` (the failing callback's owning class for bound methods, or ``__qualname__`` for free functions). For the full design rationale and the in-process accessors that back each metric see ``docs/design/v1/mp_observability/METRICS.md`` and ``docs/design/v1/mp_observability/event-bus.md`` in the source tree. Prometheus Scrape Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In **pull mode** (no ``--otlp-endpoint``), point Prometheus at the server's HTTP-frontend port — ``--http-port``, default **8080** — not at ``--prometheus-port``: .. code-block:: yaml scrape_configs: - job_name: "lmcache-mp" static_configs: - targets: [":8080"] # --http-port, NOT --prometheus-port In **push mode** (``--otlp-endpoint`` set), the server does not expose ``/metrics`` itself; scrape the OpenTelemetry Collector's Prometheus exporter instead. The bundled stack in ``examples/observability/`` wires this up for you — see :doc:`index`.