wehub-resource-sync
643 lines
27 KiB
ReStructuredText
643 lines
27 KiB
ReStructuredText
Overview
|
|
========
|
|
|
|
LMCache multiprocess (MP) mode runs LMCache as a **standalone service** that
|
|
vLLM instances connect to over ZMQ. One LMCache server per node can serve
|
|
multiple vLLM pods, providing process isolation, shared caching, and
|
|
independent resource scaling.
|
|
|
|
.. contents::
|
|
:local:
|
|
:depth: 2
|
|
|
|
Key Benefits
|
|
------------
|
|
|
|
- **Process isolation** -- LMCache and vLLM run in separate processes (or
|
|
containers), so a cache-related issue does not crash the inference engine.
|
|
- **No GIL contention or Python overhead on the inference path** -- By running
|
|
LMCache in a separate process, its Python GIL and CPU work (hashing,
|
|
memory management, L2 I/O) do not compete with vLLM's inference threads.
|
|
- **Shared caching across pods** -- Multiple vLLM instances on the same node
|
|
share a single L1 cache, maximizing KV reuse.
|
|
- **Independent resource scaling** -- Allocate CPU memory for caching
|
|
independently of GPU memory for inference.
|
|
- **Multi-tier storage (L1 + L2)** -- An L1 cache (in CPU DRAM, or an NVMe
|
|
slab via GPUDirect Storage) backed by persistent L2 storage via NIXL (GDS,
|
|
POSIX, HF3FS, and more).
|
|
- **Built-in observability** -- Prometheus metrics and a telemetry event system
|
|
out of the box.
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
- **vLLM** latest version is recommended for best compatibility
|
|
- **LMCache** latest dev branch
|
|
|
|
Server Variants
|
|
---------------
|
|
|
|
LMCache ships two server entry points:
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 30 70
|
|
|
|
* - Entry Point
|
|
- Description
|
|
* - ``lmcache server``
|
|
- **Recommended.** ZMQ + FastAPI HTTP frontend — see :doc:`http_api`.
|
|
* - ``python3 -m lmcache.v1.multiprocess.server``
|
|
- (Legacy) ZMQ-only server with no HTTP endpoints; same
|
|
``--engine-type`` / ``--supported-transfer-mode`` flags as
|
|
``lmcache server``. Prefer ``lmcache server``.
|
|
|
|
The sections below describe LMCache MP internals -- useful if you want to
|
|
understand, debug, or extend the system.
|
|
|
|
High-Level Architecture
|
|
-----------------------
|
|
|
|
.. code-block:: text
|
|
|
|
vLLM Instance(s)
|
|
|
|
|
| ZMQ (tcp)
|
|
v
|
|
MessageQueueServer (mq.py)
|
|
|
|
|
| dispatch by RequestType
|
|
v
|
|
MPCacheServer (server.py)
|
|
|
|
|
|--- TokenHasher / SessionManager
|
|
|
|
|
v
|
|
StorageManager (distributed/storage_manager.py)
|
|
|
|
|
|--- L1Manager (l1_manager.py)
|
|
| |--- L1MemoryManager (CPU DRAM) or
|
|
| | GDSL1MemoryManager (NVMe slab via cuFile / hipFile)
|
|
| |--- TTLLock per object (read/write)
|
|
|
|
|
|--- StoreController -----> L2 Adapter(s) (async L1->L2 push)
|
|
|--- PrefetchController ---> L2 Adapter(s) (async L2->L1 load)
|
|
|--- EvictionController ----> L1Manager (watermark-triggered eviction)
|
|
|
|
|
v
|
|
EventBus + OTel providers (observability)
|
|
|
|
Engine and Modules
|
|
------------------
|
|
|
|
All server entry points share the same ``MPCacheServer`` and
|
|
``StorageManager`` core. ``MPCacheServer`` is now a thin compositor:
|
|
it holds an ``MPCacheServerContext`` and a list of ``EngineModule``
|
|
instances assembled by ``_build_modules()`` (in ``server.py``)
|
|
based on ``--engine-type`` and ``--supported-transfer-mode``.
|
|
|
|
**``server.py``** -- The default ZMQ-only server. Creates an
|
|
``MPCacheServer``, assembles the engine modules
|
|
(``LookupModule`` + ``ManagementModule`` + ``LMCacheDrivenTransferModule``
|
|
and/or ``EngineDrivenTransferModule`` depending on
|
|
``--supported-transfer-mode`` — ``lmcache_driven`` or ``engine_driven`` loads
|
|
just one,
|
|
``auto`` (default) loads both — plus a CacheBlend module when
|
|
``--engine-type`` is set: ``blend`` appends ``BlendV3Module`` (the
|
|
current paged-aware implementation), and ``blend_legacy`` appends
|
|
``BlendModule`` (the original)). Starts a ``MessageQueueServer``,
|
|
registers handlers for every ``RequestType`` exposed by the loaded
|
|
modules, and blocks in a keep-alive loop.
|
|
|
|
**``modules/blend.py``** -- Defines ``BlendModule`` and ``BlendEngineV2``,
|
|
which add the original CacheBlend operations (``CB_REGISTER_KV_CACHE``,
|
|
``CB_LOOKUP_PRE_COMPUTED``, ``CB_STORE_PRE_COMPUTED``,
|
|
``CB_RETRIEVE_PRE_COMPUTED``, ``CB_STORE_FINAL`` and their V2
|
|
variants). Enables non-prefix KV cache reuse across document
|
|
paragraphs. Selected by passing ``--engine-type blend_legacy`` to
|
|
``lmcache server``.
|
|
|
|
**``modules/blend_v3.py``** -- Defines ``BlendV3Module``, the
|
|
paged-aware CacheBlend V3 pipeline that runs on the sparse-prefetch
|
|
path. Adds the V3 RPCs (``CB_REGISTER_ROPE_V3``,
|
|
``CB_UNREGISTER_ROPE_V3``, ``CB_RETRIEVE_PRE_COMPUTED_V3``,
|
|
``CB_UNIFIED_LOOKUP``) and reuses the existing
|
|
``LMCacheDrivenTransferModule`` and ``LookupModule``. Selected by
|
|
passing ``--engine-type blend`` to
|
|
``lmcache server``.
|
|
|
|
Both blend variants require ``--supported-transfer-mode`` to be
|
|
``lmcache_driven`` or ``auto`` and will refuse to load when it is
|
|
``engine_driven``.
|
|
|
|
**``http_server.py``** -- Wraps ``run_cache_server()`` (from ``server.py``)
|
|
inside a FastAPI application. Endpoints are contributed by modules under
|
|
``http_apis/`` and auto-registered via ``HTTPAPIRegistry``: ``GET /`` (basic
|
|
liveness), ``GET /healthcheck`` for Kubernetes probes, ``POST /cache/clear``
|
|
for clearing all KV cache data in L1 (CPU) memory, and ``GET /status``
|
|
for inspecting detailed internal state. The ZMQ server runs as part of the
|
|
same process, and any configured runtime plugins are spawned by
|
|
``MPRuntimePluginLauncher`` during FastAPI startup.
|
|
|
|
ZMQ Protocol
|
|
------------
|
|
|
|
Communication between vLLM and LMCache uses ZMQ (DEALER/ROUTER pattern).
|
|
|
|
**RequestType enum** (defined in ``protocols/base.py``):
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 35 20 45
|
|
|
|
* - Request Type
|
|
- Handler Type
|
|
- Description
|
|
* - ``REGISTER_KV_CACHE``
|
|
- SYNC
|
|
- Register GPU KV cache tensors for a vLLM instance.
|
|
* - ``UNREGISTER_KV_CACHE``
|
|
- SYNC
|
|
- Unregister KV cache tensors.
|
|
* - ``REGISTER_KV_CACHE_ENGINE_DRIVEN_CONTEXT``
|
|
- SYNC
|
|
- Register an engine-driven KV cache context (CPU/accelerator
|
|
workers using the PREPARE/COMMIT transfer path). Loaded only when
|
|
``--supported-transfer-mode`` is ``engine_driven`` or ``auto``.
|
|
Returns a ``RegisterEngineDrivenContextResponse`` carrying the
|
|
SHM segment name and pool size when the SHM path is in use
|
|
(empty for the pickle path).
|
|
* - ``UNREGISTER_KV_CACHE_ENGINE_DRIVEN_CONTEXT``
|
|
- SYNC
|
|
- Unregister an engine-driven KV cache context.
|
|
* - ``STORE``
|
|
- BLOCKING
|
|
- Store KV cache chunks from GPU to L1 (CPU). LMCache-driven
|
|
transfer path (CUDA IPC); loaded only when
|
|
``--supported-transfer-mode`` is ``lmcache_driven`` or ``auto``.
|
|
* - ``RETRIEVE``
|
|
- BLOCKING
|
|
- Copy KV cache chunks from L1 (CPU) back to GPU. LMCache-driven
|
|
transfer path (CUDA IPC); loaded only when
|
|
``--supported-transfer-mode`` is ``lmcache_driven`` or ``auto``.
|
|
* - ``PREPARE_STORE``
|
|
- BLOCKING
|
|
- (Engine-driven path) Worker asks the server to prepare store-side
|
|
transfer state for a key. Loaded when ``--supported-transfer-mode``
|
|
is ``engine_driven`` or ``auto``.
|
|
* - ``COMMIT_STORE``
|
|
- BLOCKING
|
|
- (Engine-driven path) Worker commits the chunk's serialized bytes
|
|
(pickle path) or releases the prepared SHM slot (SHM path) so the
|
|
server can persist into L1 storage.
|
|
* - ``PREPARE_RETRIEVE``
|
|
- BLOCKING
|
|
- (Engine-driven path) Worker asks the server to prepare the
|
|
retrieval payload for a key. The pickle path returns the bytes
|
|
inline; the SHM path returns slot info so the worker can read
|
|
from shared memory.
|
|
* - ``COMMIT_RETRIEVE``
|
|
- BLOCKING
|
|
- (Engine-driven path) Worker acknowledges retrieval completion so
|
|
the server can release the underlying read locks and reclaim any
|
|
transport state.
|
|
* - ``LOOKUP``
|
|
- BLOCKING
|
|
- Submit a prefix lookup; the prefetch job is tracked server-side by
|
|
request_id.
|
|
* - ``QUERY_PREFETCH_STATUS``
|
|
- BLOCKING
|
|
- Poll a prefetch job by request_id. Returns the loaded chunk count
|
|
when done, or ``None`` while the prefetch is still in progress.
|
|
* - ``WAIT_PREFETCH_STATUS``
|
|
- BLOCKING
|
|
- (SGLang only) Block until a prefetch job completes, then return its
|
|
loaded chunk count, or ``None`` on timeout. The blocking alternative
|
|
to polling ``QUERY_PREFETCH_STATUS``.
|
|
* - ``QUERY_PREFETCH_LOOKUP_HITS``
|
|
- BLOCKING
|
|
- Query the lookup-phase hit chunk count by request_id, before the
|
|
prefetch finishes. Returns ``None`` while the lookup is still
|
|
running.
|
|
* - ``FREE_LOOKUP_LOCKS``
|
|
- BLOCKING
|
|
- Release read locks from a cancelled lookup without doing a full
|
|
RETRIEVE.
|
|
* - ``END_SESSION``
|
|
- BLOCKING
|
|
- Remove session state for a finished request.
|
|
* - ``CLEAR``
|
|
- BLOCKING
|
|
- Clear all cached data.
|
|
* - ``GET_CHUNK_SIZE``
|
|
- SYNC
|
|
- Return the server's chunk size.
|
|
* - ``PING``
|
|
- BLOCKING
|
|
- Liveness ping; the handler always returns ``True``.
|
|
* - ``REPORT_BLOCK_ALLOCATION``
|
|
- BLOCKING
|
|
- Fire-and-forget channel for the vLLM scheduler to report GPU block
|
|
allocation events to the observability subsystem.
|
|
* - ``NOOP``
|
|
- SYNC
|
|
- Debug heartbeat -- returns a confirmation string.
|
|
* - ``CB_REGISTER_KV_CACHE``
|
|
- SYNC
|
|
- (Blend) Register CacheBlend KV buffer.
|
|
* - ``CB_UNREGISTER_KV_CACHE``
|
|
- SYNC
|
|
- (Blend) Unregister CacheBlend KV buffer.
|
|
* - ``CB_STORE_PRE_COMPUTED``
|
|
- BLOCKING
|
|
- (Blend) Store pre-computed paragraph chunks.
|
|
* - ``CB_LOOKUP_PRE_COMPUTED``
|
|
- BLOCKING
|
|
- (Blend) Lookup pre-computed paragraph chunks.
|
|
* - ``CB_RETRIEVE_PRE_COMPUTED``
|
|
- BLOCKING
|
|
- (Blend) Retrieve pre-computed paragraph chunks to GPU.
|
|
* - ``CB_STORE_FINAL``
|
|
- BLOCKING
|
|
- (Blend) Store final blended chunks.
|
|
* - ``CB_LOOKUP_PRE_COMPUTED_V2``
|
|
- BLOCKING
|
|
- (Blend V2) Lookup pre-computed chunks; returns
|
|
``CBMatchResult`` entries (with old/cur ranges and per-chunk hashes)
|
|
so the retrieve step can skip re-hashing.
|
|
* - ``CB_RETRIEVE_PRE_COMPUTED_V2``
|
|
- BLOCKING
|
|
- (Blend V2) Retrieve pre-computed chunks using the
|
|
``CBMatchResult`` list returned by ``CB_LOOKUP_PRE_COMPUTED_V2``.
|
|
* - ``CB_REGISTER_ROPE_V3``
|
|
- SYNC
|
|
- (Blend V3) Share the RoPE cos/sin cache onto a context already
|
|
registered via ``REGISTER_KV_CACHE``.
|
|
* - ``CB_UNREGISTER_ROPE_V3``
|
|
- SYNC
|
|
- (Blend V3) Drop the RoPE state (paged KV cache lives on; use
|
|
``UNREGISTER_KV_CACHE`` to release that).
|
|
* - ``CB_RETRIEVE_PRE_COMPUTED_V3``
|
|
- BLOCKING
|
|
- (Blend V3) Scatter all matched chunks (prefix- and non-prefix-hit)
|
|
into paged KV by per-token block ID; re-RoPE only the shifted subset.
|
|
* - ``CB_UNIFIED_LOOKUP``
|
|
- BLOCKING
|
|
- (Blend V3) Sole live lookup path: one RPC runs prefix + non-prefix
|
|
match, reconciles, issues one sparse-coalesced prefetch, and
|
|
classifies per-TP-rank. Returns ``CBUnifiedLookupResult`` (or
|
|
``None`` while the prefetch is still in flight).
|
|
* - ``P2P_LOOKUP_AND_LOCK``
|
|
- BLOCKING
|
|
- (P2P) Look up the given keys and read-lock the locally cached
|
|
prefix. Returns a task id which the caller passes to
|
|
``P2P_QUERY_LOOKUP_RESULTS`` to poll for the transfer addresses.
|
|
Served by ``P2PController`` (loaded unconditionally by
|
|
``_build_modules()``); whether this server also acts as a P2P
|
|
client is controlled by ``--p2p-advertise-url`` -- see
|
|
:doc:`p2p`.
|
|
* - ``P2P_QUERY_LOOKUP_RESULTS``
|
|
- BLOCKING
|
|
- (P2P) Poll the transfer addresses for a lookup task. Returns a
|
|
list of ``TransferChannelAddress`` once the lookup is complete,
|
|
or ``None`` while the lookup is still in progress or its
|
|
results have already been consumed.
|
|
* - ``P2P_UNLOCK_OBJECTS``
|
|
- BLOCKING
|
|
- (P2P) Release the read locks previously taken by
|
|
``P2P_LOOKUP_AND_LOCK`` on the given keys.
|
|
|
|
**Handler types:**
|
|
|
|
- **SYNC** -- Runs directly in the ZMQ main loop (fast, non-blocking).
|
|
- **BLOCKING** -- Dispatched to a thread pool (may involve GPU copies or I/O).
|
|
|
|
Config System
|
|
-------------
|
|
|
|
Each config module exposes a composable triple:
|
|
|
|
.. code-block:: text
|
|
|
|
(DataclassConfig, add_*_args(parser), parse_args_to_*_config(args))
|
|
|
|
``server.py:parse_args()`` composes them:
|
|
|
|
.. code-block:: python
|
|
|
|
parser = argparse.ArgumentParser(...)
|
|
add_mp_server_args(parser) # from multiprocess/config.py
|
|
# includes runtime-plugin args
|
|
# (--runtime-plugin-locations,
|
|
# --runtime-plugin-config)
|
|
add_storage_manager_args(parser) # from distributed/config.py
|
|
# which internally calls add_l2_adapters_args(parser)
|
|
add_observability_args(parser) # from mp_observability/config.py
|
|
|
|
``http_server.py`` reuses this pattern, adding
|
|
``add_http_frontend_args()`` and ``add_coordinator_args()`` for the
|
|
``lmcache server`` CLI. CacheBlend is no longer a separate entry point —
|
|
it is opted into at runtime by passing ``--engine-type`` to
|
|
``server.py`` (or ``lmcache server``). ``--engine-type blend`` appends
|
|
``BlendV3Module`` (the current paged-aware implementation), while
|
|
``--engine-type blend_legacy`` appends ``BlendModule`` (the original).
|
|
|
|
Distributed Storage
|
|
-------------------
|
|
|
|
StorageManager
|
|
~~~~~~~~~~~~~~
|
|
|
|
``lmcache/v1/distributed/storage_manager.py``
|
|
|
|
The top-level manager that wires together L1, L2, and all controllers. Key
|
|
methods:
|
|
|
|
- ``reserve_write()`` / ``finish_write()`` -- Two-phase write into L1.
|
|
- ``submit_prefetch_task()`` / ``query_prefetch_status()`` -- Async lookup +
|
|
L2 prefetch. ``query_prefetch_status()`` is non-blocking and returns
|
|
``None`` while the L2 prefetch is still in flight.
|
|
- ``wait_prefetch_status()`` -- Blocking alternative to polling
|
|
``query_prefetch_status()``: waits on a controller condition variable
|
|
until the prefetch result is published (or a timeout expires).
|
|
L1-only prefetches return immediately. Used by the
|
|
``WAIT_PREFETCH_STATUS`` RPC to avoid busy-polling on the load path.
|
|
- ``read_prefetched_results()`` / ``finish_read_prefetched()`` -- Read
|
|
prefetched data from L1 with automatic lock management.
|
|
|
|
L1Manager
|
|
~~~~~~~~~
|
|
|
|
``lmcache/v1/distributed/l1_manager.py``
|
|
|
|
Manages objects in CPU memory with a state machine:
|
|
|
|
.. code-block:: text
|
|
|
|
None --> write_locked --> ready --> read_locked
|
|
(reserve_write) (finish_write) (reserve_read)
|
|
| |
|
|
v v
|
|
evictable finish_read -> ready
|
|
|
|
Each object has two ``TTLLock`` instances (read and write) with configurable
|
|
timeouts to prevent deadlocks from crashed clients.
|
|
|
|
The underlying memory allocation is handled by one of two interchangeable
|
|
tiers selected at startup (both satisfy ``L1ManagerProtocol``):
|
|
|
|
- ``L1MemoryManager`` (default) -- pinned CPU DRAM, with lazy growth up to
|
|
``--l1-size-gb``.
|
|
- ``GDSL1MemoryManager`` -- an NVMe slab file when ``--gds-l1-path`` is set.
|
|
The bytes live on disk; reads/writes DMA directly between the GPU staging
|
|
buffer and the slab, driven by the process-global ``GDSContext``
|
|
(``gpu_connector/gds_context.py``) and dispatched from ``gpu_ops``. The DMA
|
|
backend is selected by platform via ``gpu_connector/_gds_async.py`` --
|
|
cuFile (``libcufile.so``) on NVIDIA and hipFile (``libhipfile.so``) on AMD
|
|
ROCm; see the *GDS L1 Tier* section of :doc:`configuration` for the
|
|
vendor-specific requirements. The CPU tier is disabled in this mode.
|
|
|
|
L2 Adapters
|
|
~~~~~~~~~~~
|
|
|
|
``lmcache/v1/distributed/l2_adapters/``
|
|
|
|
The ``L2AdapterInterface`` (in ``base.py``) defines three async task methods:
|
|
|
|
- ``submit_store_task(key, data)`` -- Push data to L2.
|
|
- ``submit_lookup_and_lock_task(keys)`` -- Check if keys exist in L2.
|
|
- ``submit_load_task(keys, layout_desc)`` -- Load data from L2 into L1.
|
|
|
|
The factory function ``create_l2_adapter()`` (in ``__init__.py``) uses
|
|
``isinstance()`` on the config type to instantiate the correct adapter.
|
|
|
|
New adapter types are registered via ``register_l2_adapter_type()`` in
|
|
``config.py``.
|
|
|
|
Controllers
|
|
~~~~~~~~~~~
|
|
|
|
**StoreController** (``storage_controllers/store_controller.py``):
|
|
Event-driven background thread that uses ``select.poll()`` on listener eventfd
|
|
and adapter store eventfds. When new objects appear in L1 (signaled via
|
|
``StoreListener``), it submits async store tasks to each L2 adapter based on
|
|
the ``StorePolicy``.
|
|
|
|
**EvictionController** (``storage_controllers/eviction_controller.py``):
|
|
Periodically checks L1 memory usage against the watermark threshold. When
|
|
triggered, evicts objects using the configured policy (``LRU``,
|
|
``IsolatedLRU``, or ``noop``) until usage drops below the target.
|
|
``IsolatedLRU`` evicts per ``cache_salt`` against limits registered through
|
|
the ``/quota`` HTTP endpoints; see :ref:`mp-http-quota-api`.
|
|
|
|
**PrefetchController** (``storage_controllers/prefetch_controller.py``):
|
|
Handles L2 lookup and load requests submitted by ``StorageManager`` during
|
|
``LOOKUP`` RPCs. When keys are not in L1, it queries L2 adapters and loads
|
|
found data back into L1.
|
|
|
|
Request Flows
|
|
-------------
|
|
|
|
LOOKUP Flow
|
|
~~~~~~~~~~~
|
|
|
|
.. code-block:: text
|
|
|
|
vLLM MPCacheServer StorageManager L1Manager L2 (PrefetchController)
|
|
| | | | |
|
|
|---LOOKUP(key)-------->| | | |
|
|
| |--submit_prefetch------>| | |
|
|
| | |--reserve_read----->| |
|
|
| | |<--hit_count--------| |
|
|
| | |--submit_prefetch_request--------------->|
|
|
| | | (remaining keys) |
|
|
| |--query_prefetch------->| | |
|
|
| | |--query_prefetch_result----------------->|
|
|
| |<--found_count----------| | |
|
|
|<--found_count---------| | | |
|
|
|
|
STORE Flow
|
|
~~~~~~~~~~
|
|
|
|
.. code-block:: text
|
|
|
|
vLLM MPCacheServer StorageManager L1Manager
|
|
| | | |
|
|
|---STORE(key,blocks)-->| | |
|
|
| |--reserve_write-------->| |
|
|
| | |--reserve_write---->|
|
|
| | |<--memory_objs------|
|
|
| | (GPU->CPU copy) | |
|
|
| |--finish_write--------->| |
|
|
| | |--finish_write----->|
|
|
| | | |
|
|
| | | [StoreController detects new objects]
|
|
| | | [async L1->L2 push via adapters]
|
|
|<--event_handle--------| | |
|
|
|
|
RETRIEVE Flow
|
|
~~~~~~~~~~~~~
|
|
|
|
.. code-block:: text
|
|
|
|
vLLM MPCacheServer StorageManager L1Manager
|
|
| | | |
|
|
|---RETRIEVE(key)------>| | |
|
|
| |--read_prefetched------>| |
|
|
| | |--unsafe_read------>|
|
|
| | |<--memory_objs------|
|
|
| | (CPU->GPU copy) | |
|
|
| |--finish_read_prefetch->| |
|
|
| | |--finish_read------>|
|
|
|<--event_handle--------| | |
|
|
|
|
Observability Internals
|
|
-----------------------
|
|
|
|
**EventBus** (``lmcache/v1/mp_observability/event_bus.py``) is a global
|
|
singleton initialized at server startup by ``init_observability()``.
|
|
Producers (L1Manager, StorageManager, MPCacheServer) publish ``Event``
|
|
objects to a bounded queue (``--event-bus-queue-size``, default 10000,
|
|
tail-drop on overflow). A background drain thread dispatches each
|
|
event to all registered subscribers.
|
|
|
|
**Subscribers** live under ``lmcache/v1/mp_observability/subscribers/``
|
|
and are grouped by concern: ``metrics/`` (OTel counters and lifecycle
|
|
histograms), ``logging/`` (Python logging handlers, lookup-hash JSONL),
|
|
and ``tracing/`` (OTel spans built from START/END event pairs).
|
|
``init_observability()`` registers the set selected by CLI flags
|
|
(``--disable-metrics``, ``--disable-logging``, ``--enable-tracing``).
|
|
|
|
**OTel providers** are set up via ``otel_init.py`` before subscribers
|
|
are constructed, so module-level ``get_meter()`` / ``get_tracer()``
|
|
calls bind to the real provider. Metrics are exported both to an
|
|
in-process Prometheus ``/metrics`` endpoint (``--prometheus-port``,
|
|
default 9090) and, when ``--otlp-endpoint`` is set, pushed to an OTel
|
|
collector.
|
|
|
|
How to Extend
|
|
-------------
|
|
|
|
Adding a new L2 adapter
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Create a new ``*_l2_adapter.py`` module under
|
|
``lmcache/v1/distributed/l2_adapters/`` — ``__init__.py`` auto-discovers
|
|
modules matching that suffix via ``pkgutil`` and imports them lazily on
|
|
first use, so no other files need to be modified.
|
|
|
|
1. Create a config class subclassing ``L2AdapterConfigBase`` with
|
|
``from_dict()`` and ``help()`` methods.
|
|
2. Create an adapter class implementing ``L2AdapterInterface``, and
|
|
a small factory function
|
|
``(config, l1_memory_desc) -> L2AdapterInterface``.
|
|
3. At module level, self-register both the config and the factory:
|
|
|
|
.. code-block:: python
|
|
|
|
register_l2_adapter_type("my_adapter", MyAdapterConfig)
|
|
register_l2_adapter_factory("my_adapter", _create_my_adapter)
|
|
|
|
See ``mock_l2_adapter.py`` or ``s3_l2_adapter.py`` for reference
|
|
implementations.
|
|
|
|
Adding an observability subscriber
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
1. Create a subscriber class subclassing ``EventSubscriber`` (defined
|
|
in ``lmcache/v1/mp_observability/event_bus.py``): implement
|
|
``get_subscriptions()`` to return an ``{EventType: callback}``
|
|
mapping; optionally override ``shutdown()`` for cleanup.
|
|
2. Place the class under the appropriate concern group
|
|
(``subscribers/metrics/``, ``subscribers/logging/``, or
|
|
``subscribers/tracing/``) and export it from that package's
|
|
``__init__.py``.
|
|
3. Register the subscriber in ``init_observability()``
|
|
(``lmcache/v1/mp_observability/config.py``) via
|
|
``bus.register_subscriber(...)`` inside the branch matching its
|
|
concern (metrics / logging / tracing), gated on the corresponding
|
|
CLI flag if needed.
|
|
|
|
Adding a new request type
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
1. Add a new member to ``RequestType`` in ``protocols/base.py``.
|
|
2. Create a ``ProtocolDefinition`` in the appropriate ``protocols/*.py`` file
|
|
(``engine``, ``controller``, ``observability``, ``debug``, ``blend``,
|
|
``blend_v2``, ``blend_v3``, or ``p2p``) and add the request name to that
|
|
module's ``REQUEST_NAMES``.
|
|
3. Implement the handler method on the appropriate ``EngineModule``
|
|
(e.g. ``LookupModule``, ``LMCacheDrivenTransferModule``, ``BlendV3Module``) and
|
|
expose it as a ``HandlerSpec`` from that module's ``get_handlers()``.
|
|
4. ``run_cache_server()`` registers every ``HandlerSpec`` returned by the
|
|
loaded modules via ``add_handler_helper()`` — no manual registration
|
|
step is needed.
|
|
|
|
Key Source Files
|
|
----------------
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 40 60
|
|
|
|
* - File
|
|
- Purpose
|
|
* - ``lmcache/v1/multiprocess/server.py``
|
|
- MPCacheServer + ZMQ server entry point
|
|
* - ``lmcache/v1/multiprocess/config.py``
|
|
- MPServerConfig, HTTPFrontendConfig
|
|
* - ``lmcache/v1/multiprocess/engine_context.py``
|
|
- MPCacheServerContext (shared state passed to every EngineModule)
|
|
* - ``lmcache/v1/multiprocess/engine_module.py``
|
|
- ``EngineModule`` protocol, ``HandlerSpec``, ``ThreadPoolType``
|
|
(per-module handler registration)
|
|
* - ``lmcache/v1/multiprocess/modules/``
|
|
- Engine module implementations: ``lookup.py`` (``LookupModule``),
|
|
``management.py`` (``ManagementModule``), ``lmcache_driven_transfer.py``
|
|
(``LMCacheDrivenTransferModule``), ``engine_driven_transfer.py``
|
|
(``EngineDrivenTransferModule``), ``blend.py``
|
|
(``BlendModule`` / ``BlendEngineV2``, selected by
|
|
``--engine-type blend_legacy``), and ``blend_v3.py``
|
|
(``BlendV3Module``, the paged-aware CacheBlend V3 pipeline
|
|
selected by ``--engine-type blend``).
|
|
* - ``lmcache/v1/multiprocess/http_server.py``
|
|
- FastAPI wrapper with health check and many other useful APIs
|
|
* - ``lmcache/v1/multiprocess/http_api_registry.py``
|
|
- ``HTTPAPIRegistry`` that auto-discovers routers in ``http_apis/``
|
|
* - ``lmcache/v1/multiprocess/http_apis/``
|
|
- Extensible HTTP endpoints (``/``, ``/healthcheck``,
|
|
``/cache/clear``, ``/status``)
|
|
* - ``lmcache/v1/multiprocess/mp_runtime_plugin_launcher.py``
|
|
- ``MPRuntimePluginLauncher`` that spawns runtime plugins with the
|
|
full server config serialized into environment variables
|
|
* - ``lmcache/v1/multiprocess/protocols/base.py``
|
|
- RequestType, HandlerType, ProtocolDefinition
|
|
* - ``lmcache/v1/distributed/storage_manager.py``
|
|
- StorageManager (top-level manager)
|
|
* - ``lmcache/v1/distributed/config.py``
|
|
- StorageManagerConfig hierarchy
|
|
* - ``lmcache/v1/distributed/l1_manager.py``
|
|
- L1Manager (object state machine)
|
|
* - ``lmcache/v1/distributed/l2_adapters/config.py``
|
|
- L2 adapter config registry
|
|
* - ``lmcache/v1/distributed/l2_adapters/base.py``
|
|
- L2AdapterInterface
|
|
* - ``lmcache/v1/distributed/storage_controllers/store_controller.py``
|
|
- StoreController (event-driven L1->L2)
|
|
* - ``lmcache/v1/distributed/storage_controllers/eviction_controller.py``
|
|
- EvictionController (watermark-triggered)
|
|
* - ``lmcache/v1/distributed/storage_controllers/prefetch_controller.py``
|
|
- PrefetchController (L2->L1 on miss)
|
|
* - ``lmcache/v1/mp_observability/config.py``
|
|
- ObservabilityConfig + ``init_observability()`` entry point
|
|
* - ``lmcache/v1/mp_observability/event_bus.py``
|
|
- EventBus singleton and ``EventSubscriber`` base class
|
|
* - ``lmcache/v1/mp_observability/event.py``
|
|
- ``Event`` / ``EventType`` definitions
|
|
* - ``lmcache/v1/mp_observability/otel_init.py``
|
|
- OTel metrics / tracing provider setup
|
|
* - ``lmcache/v1/mp_observability/subscribers/``
|
|
- Metrics, logging, and tracing subscribers
|
|
* - ``lmcache/v1/mp_observability/trace/``
|
|
- Trace recording (``--trace-level storage``) capture stack
|