chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,112 @@
|
||||
Architecture Overview
|
||||
=====================
|
||||
|
||||
High-Level System Architecture
|
||||
------------------------------
|
||||
|
||||
|
||||
LMCache extends an LLM inference engine (e.g., vLLM) with a multi-tier KV cache storage system spanning GPU memory, CPU memory, and disk/remote backends. The diagram below illustrates how KV cache blocks move across these layers.
|
||||
|
||||
**Multi-Tier Storage Architecture**
|
||||
|
||||
LMCache implements a hierarchical storage system with three distinct tiers:
|
||||
|
||||
* **GPU Memory**: Holds the active working set of KV caches that are currently being used by the model
|
||||
* **CPU DRAM**: Acts as a "hot cache" for recently used KV chunks, using pinned memory for efficient GPU-CPU transfers
|
||||
* **Local storage (e.g., local disk, NVMe GDS)**: Provides a large tier for local KV caching (e.g. for long documents)
|
||||
* **Remote storage (e.g., Redis, Mooncake, InfiniStore)**: Persistent storage for KV caches. Reliable but not as performant as previous tiers.
|
||||
|
||||
**Data Flow and Operations**
|
||||
|
||||
When the model generates new key-value (KV) cache chunks on the GPU, LMCache can:
|
||||
|
||||
1. **Offload overflow KV caches** from GPU to CPU DRAM, freeing precious GPU memory
|
||||
2. **Asynchronously write** KV caches from CPU to disk or remote storage using LRU eviction policies
|
||||
3. **Prefetch hot KV caches** from disk/remote storage back to CPU when needed
|
||||
4. **On-demand reuse** of cached segments by moving them from CPU back to GPU
|
||||
|
||||
This architecture enables LMCache to significantly reduce prefill delays and GPU memory pressure while maintaining high performance through intelligent cache management.
|
||||
|
||||
.. mermaid::
|
||||
:align: center
|
||||
|
||||
flowchart TB
|
||||
subgraph "LLM Engine (with LMCache Integration)"
|
||||
direction TB
|
||||
GPU["GPU Memory"]
|
||||
CPU["CPU DRAM"]
|
||||
GPU -- "Offload overflow KV" --> CPU
|
||||
CPU -- "On-demand reuse" --> GPU
|
||||
end
|
||||
Disk[(Disk Storage Backend)]
|
||||
Remote[(Remote Storage Backend)]
|
||||
CPU -- "Async write (LRU evict)" --> Disk
|
||||
CPU -- "Async upload" --> Remote
|
||||
Disk -- "Prefetch hot KV" --> CPU
|
||||
Remote -- "Fetch on reuse" --> CPU
|
||||
|
||||
Two modes
|
||||
---------
|
||||
|
||||
**Storage Mode (KV cache offloading)**
|
||||
LMCache acts as a persistent KV store, optimizing for high reuse across queries or sessions. It offloads infrequently used KV blocks from GPU memory and persists popular caches across sessions, boosting cache hit rates for "hot" content. KV caches survive beyond single inference calls and even process restarts when backed by disk or external storage.
|
||||
|
||||
|
||||
.. mermaid::
|
||||
|
||||
sequenceDiagram
|
||||
participant Main as LLM Inference Thread
|
||||
participant DiskTask as Disk Offload Task
|
||||
participant RemoteTask as Remote Offload Task
|
||||
Main->>Main: New KV chunk created (GPU memory)
|
||||
Main->>Main: Copy KV chunk to CPU buffer
|
||||
par Disk backend offload
|
||||
Main--)DiskTask: Spawn async disk write task
|
||||
DiskTask-->>DiskTask: Compress & save chunk to disk
|
||||
and Remote backend offload
|
||||
Main--)RemoteTask: Spawn async remote upload task
|
||||
RemoteTask-->>RemoteTask: Send chunk to remote store
|
||||
end
|
||||
Main-->>Main: Continue with next inference (no blocking)
|
||||
|
||||
|
||||
**Transport Mode (Prefill-decode disaggregation)**
|
||||
Focuses on accelerating distributed inference by routing KV cache data between nodes in real-time. Enables prefill-decode disaggregation where one server computes KV for prompts and delivers them to another server for generation without recomputation. Uses peer-to-peer channels with communication libraries like NIXL for low-latency, high-bandwidth transfers.
|
||||
|
||||
|
||||
Core Components
|
||||
---------------
|
||||
|
||||
**LLM Inference Engine Integration Module (Connector)**
|
||||
Integrated into the LLM engine (vLLM), the Connector taps into the paged KV memory manager. During prompt processing, it checks if token sequences were seen before:
|
||||
|
||||
* **Cache hit**: Fetches precomputed KV cache chunks from LMCache, bypassing computation
|
||||
* **Cache miss**: Model computes KV as usual, then Connector hands newly-generated KV data to LMCache for storage
|
||||
|
||||
**Cache Index (Token Database)**
|
||||
Maintains an internal index mapping token sequences to cached KV entries and their locations. Enables cross-request and cross-instance cache lookups with configurable chunking strategy (default 256 tokens) and hashing scheme.
|
||||
|
||||
**Memory Object & Allocator**
|
||||
Manages KV cache entries as MemoryObj instances using a custom memory allocator within LocalCPUBackend. Ensures pinned memory for fast GPU↔CPU transfers, NUMA-aware allocation, and interfaces with eviction policies (LRU by default).
|
||||
|
||||
**Asynchronous Offloading**
|
||||
Offloading / loading the KV cache chunks in an asynchronous manner to avoid blocking inference threads and GPU cycles.
|
||||
|
||||
**Remote Connectors**
|
||||
Plugin-based system for remote backends (Redis, Mooncake, NiXL). Uses generic RemoteBackend wrapper that delegates operations to connector implementations, supporting dynamic loading of custom backends.
|
||||
|
||||
LMCache Controller
|
||||
------------------
|
||||
|
||||
The Controller provides a management API for runtime cache operations:
|
||||
|
||||
* **Lookup**: Query cache entries for given token sequences and their locations
|
||||
* **Clear**: Purge KV cache entirely or for specific entries
|
||||
* **Compress/Decompress**: On-demand compression using CacheGen or decompression to full precision
|
||||
* **Move**: Migrate caches to specified locations for cache warming or optimization
|
||||
* **Pin/Unpin**: Mark cache entries as persistent to prevent eviction
|
||||
* **Health & Finish Checks**: Report worker health and confirm completion of async operations
|
||||
|
||||
The Controller coordinates with all LMCache workers in the system, providing centralized management for both single-instance and distributed deployments.
|
||||
|
||||
|
||||
@@ -0,0 +1,503 @@
|
||||
Extending the CLI
|
||||
=================
|
||||
|
||||
LMCache's CLI is built on a plugin-style architecture that supports
|
||||
**N-level nested subcommands** with zero-registration auto-discovery.
|
||||
This guide explains how to add commands at each level.
|
||||
|
||||
Architecture Overview
|
||||
---------------------
|
||||
|
||||
The CLI framework is composed of two core classes in
|
||||
``lmcache/cli/commands/base.py``:
|
||||
|
||||
- **BaseCommand** — Abstract base class for leaf commands (commands that
|
||||
perform actual work).
|
||||
- **CompositeCommand** — A ``BaseCommand`` subclass for commands that
|
||||
only group sub-subcommands. It auto-discovers child commands by
|
||||
scanning the package where it is defined.
|
||||
|
||||
The discovery mechanism uses ``pkgutil.iter_modules`` to scan direct
|
||||
submodules of a package, then collects all concrete ``BaseCommand``
|
||||
subclasses found in those modules. This means:
|
||||
|
||||
1. Each command is a separate ``.py`` file (or a sub-package with an
|
||||
``__init__.py``).
|
||||
2. No manual registration is needed — just create the file and it is
|
||||
picked up automatically.
|
||||
3. Utility/helper modules should be prefixed with ``_`` (e.g.
|
||||
``_helpers.py``) so they are excluded from the scan.
|
||||
|
||||
Auto-Discovery Mechanism
|
||||
------------------------
|
||||
|
||||
The CLI uses a unified ``discover_subclasses()`` utility (defined in
|
||||
``lmcache/v1/utils/subclass_discovery.py``) to locate command classes at
|
||||
every level. Understanding the two discovery entry points is key to
|
||||
extending the CLI.
|
||||
|
||||
Top-Level Command Discovery
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When the CLI starts, ``lmcache/cli/commands/__init__.py`` calls
|
||||
``discover_subclasses()`` on its own package (``lmcache.cli.commands``).
|
||||
This scans all **direct submodules** — both ``.py`` files and
|
||||
sub-packages (via their ``__init__.py``) — and collects every concrete
|
||||
``BaseCommand`` subclass found.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Simplified from lmcache/cli/commands/__init__.py
|
||||
from lmcache.v1.utils.subclass_discovery import discover_subclasses
|
||||
|
||||
ALL_COMMANDS = [
|
||||
cls()
|
||||
for cls in discover_subclasses(
|
||||
__name__, # "lmcache.cli.commands"
|
||||
BaseCommand,
|
||||
module_filter=lambda name: name != "base", # skip base.py itself
|
||||
)
|
||||
]
|
||||
|
||||
The resulting ``ALL_COMMANDS`` list is then registered with the root
|
||||
``argparse`` parser in ``main.py``. This means any new ``.py`` file (or
|
||||
sub-package) placed under ``lmcache/cli/commands/`` is automatically
|
||||
available as a top-level command — no manual imports or registration
|
||||
needed.
|
||||
|
||||
**Filtering rules for top-level discovery:**
|
||||
|
||||
- The module named ``base`` is explicitly excluded (it defines the
|
||||
abstract base classes, not a command).
|
||||
- Modules whose name starts with ``_`` are excluded by the underlying
|
||||
``pkgutil.iter_modules`` convention (they are treated as private).
|
||||
- Only **concrete** subclasses are collected — abstract classes are
|
||||
skipped.
|
||||
- Only classes **defined in** the scanned module are collected (re-exports
|
||||
are ignored via ``require_defined_in_module=True``).
|
||||
|
||||
Sub-Command Discovery (CompositeCommand)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When a ``CompositeCommand`` registers itself with the parser, its
|
||||
``register()`` method calls ``discover_subclasses()`` on the package
|
||||
where the ``CompositeCommand`` subclass is defined (i.e. its own
|
||||
``__init__.py``'s package).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Simplified from CompositeCommand.register() in base.py
|
||||
package = self.__class__.__module__ # e.g. "lmcache.cli.commands.bench"
|
||||
|
||||
for cls in discover_subclasses(
|
||||
package,
|
||||
BaseCommand,
|
||||
module_filter=lambda name: not name.startswith("_"),
|
||||
require_defined_in_module=True,
|
||||
):
|
||||
if cls is self.__class__:
|
||||
continue # skip the CompositeCommand itself
|
||||
inst = cls()
|
||||
inst.register(inner) # register as a nested subcommand
|
||||
|
||||
**Filtering rules for sub-command discovery:**
|
||||
|
||||
- Modules starting with ``_`` are excluded (use this prefix for helper
|
||||
files like ``_utils.py``).
|
||||
- The ``CompositeCommand`` class itself is skipped to avoid infinite
|
||||
recursion.
|
||||
- Only concrete ``BaseCommand`` subclasses **defined in** the scanned
|
||||
module are collected.
|
||||
- The scan is **non-recursive** (depth 1 only) — each
|
||||
``CompositeCommand`` only sees its own package's direct children.
|
||||
|
||||
**How nesting works:** If a child is itself a ``CompositeCommand``
|
||||
(defined in a sub-package's ``__init__.py``), its own ``register()``
|
||||
method will in turn scan *its* package for further children. This
|
||||
creates the recursive nesting without any special configuration.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
CLI startup
|
||||
└── __init__.py discovers ALL top-level commands
|
||||
├── ping.py → PingCommand (leaf)
|
||||
├── bench/__init__.py → BenchCommand (composite)
|
||||
│ └── BenchCommand.register() discovers:
|
||||
│ ├── server_bench/__init__.py → ServerBenchCommand (leaf)
|
||||
│ ├── l2_adapter_bench/__init__.py → L2AdapterBenchCommand (leaf)
|
||||
│ └── engine_bench/__init__.py → EngineBenchCommand (leaf)
|
||||
└── tool/__init__.py → ToolCommand (composite)
|
||||
└── ToolCommand.register() discovers:
|
||||
└── cache_simulator/__init__.py → CacheSimulatorCommand (composite)
|
||||
└── CacheSimulatorCommand.register() discovers:
|
||||
├── simulate_command.py → SimulateCommand (leaf)
|
||||
├── sweep_command.py → SweepCommand (leaf)
|
||||
└── gen_dataset_command.py → GenDatasetCommand (leaf)
|
||||
|
||||
Directory Layout
|
||||
----------------
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
lmcache/cli/commands/
|
||||
├── __init__.py # Top-level discovery (scans this package)
|
||||
├── base.py # BaseCommand & CompositeCommand
|
||||
├── ping.py # Level-1 leaf command
|
||||
├── server.py # Level-1 leaf command
|
||||
├── quota/ # Level-2 composite command
|
||||
│ ├── __init__.py # QuotaCommand(CompositeCommand)
|
||||
│ ├── _helpers.py # Utility (excluded from scan by _ prefix)
|
||||
│ ├── get_command.py # Level-2 leaf: ``lmcache quota get``
|
||||
│ ├── set_command.py # Level-2 leaf: ``lmcache quota set``
|
||||
│ └── ...
|
||||
└── tool/ # Level-2 composite command
|
||||
├── __init__.py # ToolCommand(CompositeCommand)
|
||||
└── cache_simulator/ # Level-3 composite command
|
||||
├── __init__.py # CacheSimulatorCommand(CompositeCommand)
|
||||
├── simulate_command.py # Level-3 leaf: ``lmcache tool cache-simulator simulate``
|
||||
└── sweep_command.py # Level-3 leaf: ``lmcache tool cache-simulator sweep``
|
||||
|
||||
Key Rules
|
||||
^^^^^^^^^
|
||||
|
||||
- A ``CompositeCommand`` subclass **must** be defined in the
|
||||
``__init__.py`` of its package.
|
||||
- Modules starting with ``_`` are **excluded** from auto-discovery
|
||||
(use this for helpers, utilities, internal logic).
|
||||
- Each leaf command file should contain exactly one concrete
|
||||
``BaseCommand`` subclass.
|
||||
- The scan is **non-recursive** — each ``CompositeCommand`` only scans
|
||||
its own package's direct submodules.
|
||||
|
||||
Level 1: Adding a Top-Level Command
|
||||
------------------------------------
|
||||
|
||||
A top-level command appears directly under ``lmcache <command>``.
|
||||
|
||||
**Step 1**: Create a new file under ``lmcache/cli/commands/``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/hello.py
|
||||
"""``lmcache hello`` — a simple greeting command."""
|
||||
|
||||
import argparse
|
||||
|
||||
from lmcache.cli.commands.base import BaseCommand
|
||||
|
||||
|
||||
class HelloCommand(BaseCommand):
|
||||
"""Print a greeting message."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "hello"
|
||||
|
||||
def help(self) -> str:
|
||||
return "Print a greeting message."
|
||||
|
||||
def add_arguments(self, parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--name", default="World", help="Who to greet.")
|
||||
|
||||
def execute(self, args: argparse.Namespace) -> None:
|
||||
metrics = self.create_metrics("Hello", args)
|
||||
metrics.add("greeting", "Greeting", f"Hello, {args.name}!")
|
||||
metrics.emit()
|
||||
|
||||
**Step 2**: Done! The command is automatically discovered. Test it:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache hello --name LMCache
|
||||
|
||||
.. note::
|
||||
|
||||
This works because the top-level ``lmcache/cli/commands/__init__.py``
|
||||
calls ``discover_subclasses`` on its own package at import time. It
|
||||
uses ``pkgutil.iter_modules`` to find all direct submodules (files and
|
||||
sub-packages), imports each one, and collects every concrete
|
||||
``BaseCommand`` subclass. The resulting list is stored in
|
||||
``ALL_COMMANDS`` and registered with the argument parser in
|
||||
``main.py``. So adding a new ``.py`` file with a ``BaseCommand``
|
||||
subclass is all that is needed — no edits to any other file.
|
||||
|
||||
Level 2: Adding a Subcommand Group
|
||||
-----------------------------------
|
||||
|
||||
A subcommand group appears as ``lmcache <group> <subcommand>``.
|
||||
|
||||
**Step 1**: Create a package directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir lmcache/cli/commands/mygroup/
|
||||
|
||||
**Step 2**: Define the ``CompositeCommand`` in ``__init__.py``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/mygroup/__init__.py
|
||||
"""``lmcache mygroup`` command group.
|
||||
|
||||
Sub-subcommands are auto-discovered from modules in this package.
|
||||
"""
|
||||
|
||||
from lmcache.cli.commands.base import CompositeCommand
|
||||
|
||||
|
||||
class MyGroupCommand(CompositeCommand):
|
||||
"""Command group for my custom operations."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "mygroup"
|
||||
|
||||
def help(self) -> str:
|
||||
return "My custom command group."
|
||||
|
||||
**Step 3**: Add leaf subcommands as separate files:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/mygroup/foo_command.py
|
||||
"""``lmcache mygroup foo`` — do something."""
|
||||
|
||||
import argparse
|
||||
|
||||
from lmcache.cli.commands.base import BaseCommand
|
||||
|
||||
|
||||
class FooCommand(BaseCommand):
|
||||
"""Execute the foo action."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "foo"
|
||||
|
||||
def help(self) -> str:
|
||||
return "Execute the foo action."
|
||||
|
||||
def add_arguments(self, parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--value", type=int, required=True)
|
||||
|
||||
def execute(self, args: argparse.Namespace) -> None:
|
||||
metrics = self.create_metrics("Foo Result", args)
|
||||
metrics.add("result", "Result", args.value * 2)
|
||||
metrics.emit()
|
||||
|
||||
**Step 4**: (Optional) Add helper modules with ``_`` prefix:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/mygroup/_utils.py
|
||||
"""Internal utilities for mygroup commands (not auto-discovered)."""
|
||||
|
||||
def compute_something(x: int) -> int:
|
||||
return x * 42
|
||||
|
||||
**Result**:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache mygroup foo --value 5
|
||||
|
||||
Level 2: Adding a Subcommand to an Existing Group
|
||||
---------------------------------------------------
|
||||
|
||||
If a ``CompositeCommand`` group already exists (e.g. ``bench``, ``quota``,
|
||||
``trace``), you can extend it by simply adding **one new file** — no other
|
||||
changes are required.
|
||||
|
||||
For example, to add a new ``lmcache bench l2`` subcommand under the
|
||||
existing ``bench`` group:
|
||||
|
||||
**Step 1**: Create a single file (or sub-package) in the existing group's
|
||||
package directory:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/bench/l2_adapter_bench/__init__.py
|
||||
"""``lmcache bench l2`` subpackage."""
|
||||
|
||||
import argparse
|
||||
|
||||
from lmcache.cli.commands.base import BaseCommand
|
||||
|
||||
|
||||
class L2AdapterBenchCommand(BaseCommand):
|
||||
"""Benchmark an L2 adapter (store / lookup / load)."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "l2"
|
||||
|
||||
def help(self) -> str:
|
||||
return "Benchmark an L2 adapter (store / lookup / load)."
|
||||
|
||||
def add_arguments(self, parser: argparse.ArgumentParser) -> None:
|
||||
from lmcache.cli.commands.bench.l2_adapter_bench.command import (
|
||||
add_l2_arguments,
|
||||
)
|
||||
add_l2_arguments(parser)
|
||||
|
||||
def execute(self, args: argparse.Namespace) -> None:
|
||||
from lmcache.cli.commands.bench.l2_adapter_bench.command import (
|
||||
run_l2_adapter_bench,
|
||||
)
|
||||
run_l2_adapter_bench(self, args)
|
||||
|
||||
**Step 2**: Done! The parent ``CompositeCommand`` (``BenchCommand``)
|
||||
auto-discovers the new subcommand at startup. No registration code, no
|
||||
imports to add, no ``__init__.py`` edits in the parent.
|
||||
|
||||
.. note::
|
||||
|
||||
This works because ``CompositeCommand.register()`` scans all direct
|
||||
submodules of its package each time the CLI starts. A new file (or
|
||||
sub-package) is automatically picked up as long as:
|
||||
|
||||
- It does **not** start with ``_``.
|
||||
- It contains a concrete ``BaseCommand`` subclass.
|
||||
|
||||
Level N: Arbitrary Nesting
|
||||
--------------------------
|
||||
|
||||
The framework supports **unlimited nesting depth**. Each level follows
|
||||
the same pattern: a ``CompositeCommand`` in a package's ``__init__.py``
|
||||
auto-discovers its children.
|
||||
|
||||
**Example**: Adding a 3rd level under ``lmcache mygroup``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
mkdir lmcache/cli/commands/mygroup/nested/
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/mygroup/nested/__init__.py
|
||||
"""``lmcache mygroup nested`` — a nested command group."""
|
||||
|
||||
from lmcache.cli.commands.base import CompositeCommand
|
||||
|
||||
|
||||
class NestedCommand(CompositeCommand):
|
||||
"""Nested subcommand group."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "nested"
|
||||
|
||||
def help(self) -> str:
|
||||
return "A nested command group under mygroup."
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/cli/commands/mygroup/nested/bar_command.py
|
||||
"""``lmcache mygroup nested bar`` — a deeply nested command."""
|
||||
|
||||
import argparse
|
||||
|
||||
from lmcache.cli.commands.base import BaseCommand
|
||||
|
||||
|
||||
class BarCommand(BaseCommand):
|
||||
"""Execute the bar action at level 3."""
|
||||
|
||||
def name(self) -> str:
|
||||
return "bar"
|
||||
|
||||
def help(self) -> str:
|
||||
return "Execute the bar action."
|
||||
|
||||
def add_arguments(self, parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--msg", default="deep")
|
||||
|
||||
def execute(self, args: argparse.Namespace) -> None:
|
||||
metrics = self.create_metrics("Bar Result", args)
|
||||
metrics.add("message", "Message", args.msg)
|
||||
metrics.emit()
|
||||
|
||||
**Result**:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache mygroup nested bar --msg "hello from level 3"
|
||||
|
||||
You can continue nesting indefinitely by repeating this pattern.
|
||||
|
||||
Real-World Example
|
||||
------------------
|
||||
|
||||
The existing ``lmcache tool cache-simulator simulate`` command
|
||||
demonstrates 3-level nesting:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
lmcache tool cache-simulator simulate
|
||||
│ │ │ └── Level-3 leaf (SimulateCommand in simulate_command.py)
|
||||
│ │ └── Level-2 composite (CacheSimulatorCommand in cache_simulator/__init__.py)
|
||||
│ └── Level-1 composite (ToolCommand in tool/__init__.py)
|
||||
└── CLI entry point
|
||||
|
||||
Using the Metrics System
|
||||
------------------------
|
||||
|
||||
The metrics system uses a **handler + formatter** architecture:
|
||||
|
||||
- **Metrics** — the collector. Holds sections and entries.
|
||||
- **Handler** — the destination (stdout, file, etc.).
|
||||
- **Formatter** — the rendering (ASCII table, JSON, etc.).
|
||||
|
||||
``BaseCommand.create_metrics()`` sets up default handlers automatically, so
|
||||
command authors just build metrics and call ``emit()``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def execute(self, args: argparse.Namespace) -> None:
|
||||
# create_metrics() auto-registers:
|
||||
# - StreamHandler → stdout (formatter chosen by --format, default: terminal)
|
||||
# - FileHandler → if --output is set (same format as --format)
|
||||
metrics = self.create_metrics("Bench KV Cache Result", args)
|
||||
|
||||
# Create named sections
|
||||
metrics.add_section("ops", "Operations (ops/s)")
|
||||
metrics["ops"].add("store", "Store", 41.3)
|
||||
metrics["ops"].add("retrieve", "Retrieve", 127.3)
|
||||
|
||||
# Top-level metrics (no section header)
|
||||
metrics.add("status", "Status", "OK")
|
||||
|
||||
# Trigger all handlers
|
||||
metrics.emit()
|
||||
|
||||
The ``--format`` and ``--output`` flags are added automatically by
|
||||
``BaseCommand.register()`` — subcommands do not need to add them manually.
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 15 25 60
|
||||
|
||||
* - Level
|
||||
- Pattern
|
||||
- How to add
|
||||
* - 1 (top)
|
||||
- Single ``.py`` file
|
||||
- Create ``lmcache/cli/commands/<name>.py`` with a ``BaseCommand``
|
||||
subclass.
|
||||
* - 2+
|
||||
- Package directory
|
||||
- Create ``lmcache/cli/commands/<group>/__init__.py`` with a
|
||||
``CompositeCommand`` subclass, then add leaf commands as sibling
|
||||
``.py`` files.
|
||||
* - N (any)
|
||||
- Nested package
|
||||
- Same as level 2, but inside an existing composite command's
|
||||
package. Each ``CompositeCommand`` scans only its own direct
|
||||
submodules.
|
||||
|
||||
.. tip::
|
||||
|
||||
- Prefix helper/utility modules with ``_`` to exclude them from
|
||||
auto-discovery.
|
||||
- Each ``CompositeCommand`` must be defined in its package's
|
||||
``__init__.py``.
|
||||
- The ``name()`` method determines the CLI token (e.g. ``"foo"``
|
||||
becomes ``lmcache ... foo``).
|
||||
@@ -0,0 +1,204 @@
|
||||
Contributing Guide
|
||||
==================
|
||||
|
||||
Thank you for your interest in contributing to LMCache! We welcome and accept all kinds of contributions, no matter how small or large. There are several ways you can contribute to the project:
|
||||
|
||||
- Identify and report any issues or bugs
|
||||
- Request or add support for a new model
|
||||
- Suggest or implement new features
|
||||
- Improve documentation or contribute a how-to guide
|
||||
|
||||
A comprehensive list of good first issues can be found in the issue `[Onboarding 2026] Good first issues — start contributing to LMCache in one PR <https://github.com/LMCache/LMCache/issues/3372>`_.
|
||||
|
||||
If you'd like to support our community further, then answering queries, offering PR reviews, and assisting others are also impactful ways to contribute and take LMCache further.
|
||||
|
||||
Finally, you can support us by raising awareness about LMCache. Feel free to share our blog posts, check out our handle on X at `LMCache <https://x.com/lmcache>`_ and see the latest of what we are up to. If using LMCache helped your project or product in any way, you can simply offer your appreciation by starring our repository!
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
See the `LICENSE <https://github.com/LMCache/LMCache/blob/dev/LICENSE>`_ file for details.
|
||||
|
||||
Code of Conduct
|
||||
---------------
|
||||
|
||||
This project adheres to the `Code of Conduct <https://github.com/LMCache/LMCache/blob/dev/CODE_OF_CONDUCT.md>`_. By participating, you are expected to uphold this code.
|
||||
|
||||
Contribution Guidelines
|
||||
-----------------------
|
||||
|
||||
Help on open source projects is always welcome and there is always something that can be improved. For example, documentation (like the text you are reading now) can always use improvement, code can always be clarified, variables or functions can always be renamed or commented on, and there is always a need for more test coverage. If you see something that you think should be fixed, take ownership! Here is how you get started.
|
||||
|
||||
How Can I Contribute?
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When contributing, it's useful to start by looking at `issues <https://github.com/LMCache/LMCache/issues>`_. After picking up an issue, writing code, or updating a document, make a pull request and your work will be reviewed and merged. If you're adding a new feature or find a bug, it's best to `write an issue <https://github.com/LMCache/LMCache/issues/new>`_ first to discuss it with maintainers.
|
||||
|
||||
If you discover a security vulnerability, please follow the instructions in the `Security doc <https://github.com/LMCache/LMCache/blob/dev/SECURITY.md>`_.
|
||||
|
||||
To contribute to this repo, you'll use the Fork and Pull model common in many open source repositories. For details on this process, check out `The GitHub Workflow Guide <https://github.com/kubernetes/community/blob/master/contributors/guide/github-workflow.md>`_ from Kubernetes. In short:
|
||||
|
||||
- Fork the repository
|
||||
- Create a branch
|
||||
- Run code style checks and fix any issues
|
||||
- Run unit tests and fix any broken tests
|
||||
- Submit a pull request with detailed descriptions
|
||||
|
||||
When your contribution is ready, you can create a pull request. Pull requests are often referred to as "PRs". In general, we follow the standard `GitHub pull request <https://help.github.com/en/articles/about-pull-requests>`_ process. Follow the template to provide details about your pull request to the maintainers.
|
||||
|
||||
Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
|
||||
|
||||
- [Bugfix] for bug fixes
|
||||
- [Build] for build fixes and improvements
|
||||
- [CI] for continuous integration fixes and iimprovements
|
||||
- [Core] for changes in the core LMCache logic (e.g., ``LMCacheEngine``, ``Backend`` etc.)
|
||||
- [Doc] for documentation fixes and improvements
|
||||
- [Misc] for PRs that do not fit the above categories. Please use this sparingly
|
||||
- [Model] for adding a new model or improving an existing model. Model name should appear in the title
|
||||
- [Test] for unit tests
|
||||
|
||||
.. note::
|
||||
|
||||
If the PR spans more than one category, please include all relevant prefixes
|
||||
|
||||
It's best to break your contribution into smaller PRs with incremental changes, and include a good description of the changes. We require new unit tests to be contributed with any new functionality added and docs if user facing changes.
|
||||
|
||||
Before sending pull requests, make sure your changes pass code quality checks and unit tests. These checks will run when the pull request builds. Alternatively, you can run the checks manually on your local machine `as specified in Development <#development>`_ .
|
||||
|
||||
DCO and Signed-off-by
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When contributing changes to the project, you must agree to the `DCO <https://github.com/LMCache/LMCache/blob/dev/DCO>`_. Commits must include a :code:`Signed-off-by` header which certifies agreement with the terms of the `DCO <https://github.com/LMCache/LMCache/blob/dev/DCO>`_.
|
||||
|
||||
.. note::
|
||||
|
||||
Using :code:`-s` or :code:`--signoff` flag with :code:`git commit` will automatically add this header. The flags can also be used with :code:`--amend` if you forget to sign your last commit.
|
||||
|
||||
Code Review
|
||||
^^^^^^^^^^^
|
||||
|
||||
Once you've created a pull request, maintainers will review your code and may make suggestions to fix before merging. It will be easier for your pull request to receive reviews if you consider the criteria the reviewers follow while working. Remember to:
|
||||
|
||||
- Document the code well to help future contributors and maintainers
|
||||
- Follow the project coding conventions for consistency and best practices
|
||||
- Include sufficient tests to maintain robustness of the code
|
||||
- Add or update documentation in :code:`/docs` if PR modifies user facing behavior to help people using LMCache
|
||||
- Run coding style checks to ensure consistency and correctness
|
||||
- Run tests locally and ensure they pass and don't break the existing code base
|
||||
- Write detailed commit messages to help future contributors and maintainers
|
||||
- Break large changes into a logical series of smaller patches, which are easy to understand individually and combine to solve a broader issue
|
||||
|
||||
.. note::
|
||||
|
||||
Maintainers will perform "squash and merge" actions on PRs in this repo, so it doesn't matter how many commits your PR has, as they will end up being a single commit after merging.
|
||||
|
||||
Development
|
||||
-----------
|
||||
|
||||
Set up your dev environment
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The following prerequisites are required:
|
||||
|
||||
- OS: Linux
|
||||
- GPU: NVIDIA compute capability 7.0+ (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
|
||||
- CUDA 12.8+
|
||||
|
||||
The following tools are required:
|
||||
|
||||
- `git <https://git-scm.com>`_
|
||||
- `python <https://www.python.org>`_ (v3.10 -- v3.13)
|
||||
- `pip <https://pypi.org/project/pip/>`_ (v23.0+)
|
||||
|
||||
The first step is to install the necessary Python packages required for development. The commands to do this are as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Equivalent to pip install -r requirements/common.txt
|
||||
pip install -e .
|
||||
|
||||
pip install -r requirements/lint.txt
|
||||
pip install -r requirements/test.txt
|
||||
|
||||
Before pushing changes to GitHub, you need to run the coding style checks and unit tests as shown below.
|
||||
|
||||
Coding style
|
||||
^^^^^^^^^^^^
|
||||
|
||||
LMCache follows the Python `pep8 <https://peps.python.org/pep-0008/>`_ coding style for Python and `Google C++ style guide <https://google.github.io/styleguide/cppguide.html>`_ for C++. We use the following tools to enforce the coding style:
|
||||
|
||||
- Python linting and formatting: `Ruff <https://docs.astral.sh/ruff/>`_, and `isort <https://pycqa.github.io/isort/>`_
|
||||
- Python static code checking: `mypy <https://github.com/python/mypy>`_
|
||||
- Spell checking: `codespell <https://github.com/codespell-project/codespell>`_
|
||||
- C++ formatting: `clang-format <https://clang.llvm.org/docs/ClangFormat.html>`_
|
||||
|
||||
The tools are managed by `pre-commit <https://pre-commit.com/>`_. It is installed as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install -r requirements/lint.txt
|
||||
pre-commit install
|
||||
|
||||
It will run automatically when you add a commit. You can also run it manually on all files with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pre-commit run --all-files
|
||||
|
||||
.. note::
|
||||
|
||||
For all new code added, please write docstrings in `sphinx-doc format <https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html>`_.
|
||||
|
||||
Unit tests
|
||||
^^^^^^^^^^
|
||||
|
||||
When making changes, run the tests before pushing the changes. Running unit tests ensures your contributions do not break existing code. We use the `pytest <https://docs.pytest.org/>`_ framework to run unit tests. The framework is setup to run all files in the `tests <https://github.com/LMCache/LMCache/tree/dev/tests>`_ directory which have a prefix or posfix of "test".
|
||||
|
||||
Running unit tests is as simple as:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pytest
|
||||
|
||||
.. note::
|
||||
|
||||
``vLLM`` (``pip install vllm``) and the dependencies in ``requirements/test.txt`` need to be installed prior to running unit tests.
|
||||
|
||||
By default, all tests found within the tests directory are run. However, specific unit tests can run by passing filenames, classes and/or methods to `pytest`. The following example invokes a single test method "test_lm_connector" that is declared in the "tests/test_connector.py" file:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pytest tests/test_connector.py::test_lm_connector
|
||||
|
||||
.. note::
|
||||
|
||||
Some unit tests require a NVIDIA GPU. This means that on a non Linux NVIDIA GPU system, the full suite of tests will not be run (tests requiring CUDA will be skipped). The Buildkite continuous integration (CI) system executes a full run of all the tests.
|
||||
|
||||
.. note::
|
||||
|
||||
The `NVIDIA Inference Xfer Library (NIXL) <https://github.com/ai-dynamo/nixl>`_ unit tests require NIXL to be to be installed. This is not installed by LMCache by and therefore requires you to install it separately. Please follow the details in the NIXL GitHub repo to install.
|
||||
If the NIXL package is not installed, the NIXL unit tests are skipped.
|
||||
|
||||
Building the docs
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
Install the dependencies:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install -r requirements/docs.txt
|
||||
|
||||
After that, you can build the docs (from :code:`docs/` directory) using `make`:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
make clean
|
||||
make html
|
||||
|
||||
Serve docs page locally at http://localhost:8000: :code:`python -m http.server -d build/html/`
|
||||
|
||||
Thank You
|
||||
---------
|
||||
|
||||
Thank you for your contribution to LMCache and making it a better, and accessible framework for all.
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
Extending the HTTP API
|
||||
======================
|
||||
|
||||
You can add new endpoints to the ``lmcache server`` HTTP frontend **without
|
||||
modifying any existing code**. An endpoint is just a Python module placed in
|
||||
``lmcache/v1/multiprocess/http_apis/`` that exposes a FastAPI ``APIRouter``;
|
||||
``HTTPAPIRegistry`` auto-discovers and mounts it at startup -- the same
|
||||
zero-modification pattern used by the :doc:`L2 adapters
|
||||
</mp/l2_storage/index>`.
|
||||
|
||||
How discovery works
|
||||
-------------------
|
||||
|
||||
At startup, ``http_server.py`` hands the FastAPI app to ``HTTPAPIRegistry``
|
||||
(``lmcache/v1/multiprocess/http_api_registry.py``), which scans the
|
||||
``http_apis/`` directory with ``pkgutil``, imports every module whose name ends
|
||||
with ``_api``, and includes any module-level ``router``. The built-in modules
|
||||
follow this pattern:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 28 20 12 40
|
||||
|
||||
* - Module
|
||||
- Endpoint
|
||||
- Method
|
||||
- Description
|
||||
* - ``info_api.py``
|
||||
- ``/``
|
||||
- GET
|
||||
- Basic liveness check
|
||||
* - ``info_api.py``
|
||||
- ``/healthcheck``
|
||||
- GET
|
||||
- Kubernetes probe endpoint
|
||||
* - ``cache_api.py``
|
||||
- ``/cache/clear``
|
||||
- POST
|
||||
- Force-clear the L1 cache
|
||||
* - ``info_api.py``
|
||||
- ``/status``
|
||||
- GET
|
||||
- Internal status report
|
||||
|
||||
Adding an endpoint
|
||||
------------------
|
||||
|
||||
Create a file in ``lmcache/v1/multiprocess/http_apis/`` whose name ends with
|
||||
``_api.py`` and expose a ``router``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/v1/multiprocess/http_apis/metrics_api.py
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def metrics(request: Request):
|
||||
"""Return cache hit/miss metrics."""
|
||||
engine = getattr(request.app.state, "engine", None)
|
||||
if engine is None:
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={"error": "engine not initialized"},
|
||||
)
|
||||
return {"hits": 42, "misses": 7}
|
||||
|
||||
That's it -- ``HTTPAPIRegistry`` discovers and mounts it on the next server
|
||||
startup; no other file needs to change.
|
||||
|
||||
Module contract
|
||||
---------------
|
||||
|
||||
An API module **must**:
|
||||
|
||||
- live in ``lmcache/v1/multiprocess/http_apis/`` with a filename ending in
|
||||
``_api.py``;
|
||||
- expose a module-level ``router`` of type ``fastapi.APIRouter``.
|
||||
|
||||
An API module **should**:
|
||||
|
||||
- guard against uninitialized state by checking ``request.app.state.engine``
|
||||
and returning ``503`` when it is ``None``;
|
||||
- use ``lmcache.logging.init_logger(__name__)`` for logging;
|
||||
- use ``async`` handlers and avoid blocking I/O.
|
||||
|
||||
An API module **must not** import or mutate the ``app`` object from
|
||||
``http_server.py``.
|
||||
|
||||
Accessing shared state
|
||||
----------------------
|
||||
|
||||
``app.state`` is the shared context populated during server startup. Reach it
|
||||
through the request object:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@router.get("/my-endpoint")
|
||||
async def my_endpoint(request: Request):
|
||||
engine = request.app.state.engine # main cache engine
|
||||
zmq_server = request.app.state.zmq_server # underlying ZMQ server
|
||||
...
|
||||
|
||||
For the full design rationale see
|
||||
``docs/design/v1/multiprocess/http_api_extension.md`` in the source tree.
|
||||
@@ -0,0 +1,686 @@
|
||||
Adding Native Backends
|
||||
======================
|
||||
|
||||
.. _native-connectors-overview:
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Native connectors are high-performance C++ storage backends that integrate with LMCache
|
||||
through pybind11. They work in **both** LMCache operating modes:
|
||||
|
||||
- **Non-MP mode** (single process): via ``ConnectorClientBase`` (asyncio integration)
|
||||
- **MP mode** (multiprocess): via ``NativeConnectorL2Adapter`` (L2 adapter interface)
|
||||
|
||||
Write the connector once, get both modes for free.
|
||||
|
||||
The framework lives in ``csrc/storage_backends/`` with the Redis RESP connector as the
|
||||
reference implementation.
|
||||
|
||||
Architecture
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Non-MP mode:
|
||||
CacheEngine -> RemoteBackend -> ConnectorClientBase -> native client (C++)
|
||||
(asyncio event loop)
|
||||
|
||||
MP mode:
|
||||
StoreController / PrefetchController
|
||||
|
|
||||
NativeConnectorL2Adapter (Python bridge)
|
||||
+-- 3 eventfds (store, lookup, load)
|
||||
+-- completion demux thread
|
||||
+-- ObjectKey <-> string serialization
|
||||
+-- client-side lock tracking
|
||||
|
|
||||
native client (C++)
|
||||
+-- 1 eventfd, worker threads, GIL-free I/O
|
||||
|
||||
**Design principles:**
|
||||
|
||||
1. **GIL release** at the pybind layer for true concurrency between native threads
|
||||
2. **Batching with tiling**: work for a batched request is split evenly among threads
|
||||
3. **eventfd-based completions**: the kernel wakes Python -- no polling
|
||||
4. **Non-blocking submission**: submission queue / completion queue architecture
|
||||
|
||||
|
||||
Step 1: C++ Connector
|
||||
---------------------
|
||||
|
||||
Create your connector directory (e.g., ``csrc/storage_backends/mybackend/``) and
|
||||
inherit from ``ConnectorBase<YourConnectionType>``. You need to override 4 required methods
|
||||
(and optionally ``do_single_delete`` to support eviction).
|
||||
|
||||
**connector.h:**
|
||||
|
||||
.. code-block:: cpp
|
||||
|
||||
// csrc/storage_backends/mybackend/connector.h
|
||||
#pragma once
|
||||
#include "../connector_base.h"
|
||||
|
||||
namespace lmcache {
|
||||
namespace connector {
|
||||
|
||||
// Per-thread connection state
|
||||
struct MyConn {
|
||||
int fd = -1;
|
||||
// your connection fields
|
||||
};
|
||||
|
||||
class MyConnector : public ConnectorBase<MyConn> {
|
||||
public:
|
||||
MyConnector(std::string host, int port, int num_workers)
|
||||
: ConnectorBase(num_workers), host_(host), port_(port) {
|
||||
start_workers(); // IMPORTANT: call at END of constructor
|
||||
}
|
||||
|
||||
protected:
|
||||
// 1. Create a connection (called once per worker thread)
|
||||
MyConn create_connection() override {
|
||||
MyConn conn;
|
||||
// connect to server...
|
||||
return conn;
|
||||
}
|
||||
|
||||
// 2. GET: read value for key into buf
|
||||
void do_single_get(MyConn& conn, const std::string& key,
|
||||
void* buf, size_t len,
|
||||
size_t chunk_size) override {
|
||||
// send GET command, recv response into buf
|
||||
}
|
||||
|
||||
// 3. SET: write data from buf under key
|
||||
void do_single_set(MyConn& conn, const std::string& key,
|
||||
const void* buf, size_t len,
|
||||
size_t chunk_size) override {
|
||||
// send SET command with data from buf
|
||||
}
|
||||
|
||||
// 4. EXISTS: check if key exists
|
||||
bool do_single_exists(MyConn& conn,
|
||||
const std::string& key) override {
|
||||
// send EXISTS, return true/false
|
||||
}
|
||||
|
||||
// 5. DELETE: remove key (optional, has default no-op)
|
||||
bool do_single_delete(MyConn& conn,
|
||||
const std::string& key) override {
|
||||
// send DELETE, return true if deleted, false if not found
|
||||
}
|
||||
|
||||
// Optional: clean shutdown
|
||||
void shutdown_connections() override {
|
||||
// close sockets, free resources
|
||||
}
|
||||
|
||||
private:
|
||||
std::string host_;
|
||||
int port_;
|
||||
};
|
||||
|
||||
} // namespace connector
|
||||
} // namespace lmcache
|
||||
|
||||
**What ConnectorBase gives you for free:**
|
||||
|
||||
- Worker thread pool with per-thread connections
|
||||
- Submission queue (lock-free enqueue) and completion queue
|
||||
- Automatic tiling: batch operations are split across workers
|
||||
- eventfd signaling on completion (kernel wakes Python)
|
||||
- Graceful shutdown (stop flag, drain, join)
|
||||
|
||||
.. important::
|
||||
Always call ``start_workers()`` at the **end** of your derived constructor,
|
||||
after all member variables are initialized. Worker threads call
|
||||
``create_connection()`` immediately, so the object must be fully constructed.
|
||||
|
||||
**Reference:** ``csrc/storage_backends/redis/connector.h`` and ``connector.cpp``
|
||||
|
||||
|
||||
Step 2: Pybind Module
|
||||
---------------------
|
||||
|
||||
Use the ``LMCACHE_BIND_CONNECTOR_METHODS`` macro, which binds all 7 methods
|
||||
(``event_fd``, ``submit_batch_get/set/exists/delete``, ``drain_completions``, ``close``)
|
||||
with proper GIL release and Python buffer protocol handling.
|
||||
|
||||
.. code-block:: cpp
|
||||
|
||||
// csrc/storage_backends/mybackend/pybind.cpp
|
||||
#include <pybind11/pybind11.h>
|
||||
#include "../connector_pybind_utils.h"
|
||||
#include "connector.h"
|
||||
|
||||
namespace py = pybind11;
|
||||
|
||||
PYBIND11_MODULE(lmcache_mybackend, m) {
|
||||
py::class_<lmcache::connector::MyConnector>(m, "LMCacheMyBackendClient")
|
||||
.def(py::init<std::string, int, int>(),
|
||||
py::arg("host"), py::arg("port"),
|
||||
py::arg("num_workers"))
|
||||
LMCACHE_BIND_CONNECTOR_METHODS(
|
||||
lmcache::connector::MyConnector);
|
||||
}
|
||||
|
||||
The pybind utilities automatically:
|
||||
|
||||
- Extract buffer pointers from Python ``memoryview`` objects under the GIL
|
||||
- Release the GIL before calling into C++
|
||||
- Convert C++ ``Completion`` structs to Python tuples ``(future_id, ok, error, result_bools)``
|
||||
|
||||
**Reference:** ``csrc/storage_backends/redis/pybind.cpp``
|
||||
|
||||
|
||||
Step 3: Build System
|
||||
--------------------
|
||||
|
||||
Register your C++ sources in ``setup.py`` alongside the existing Redis extension:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# In cuda_extension() and rocm_extension():
|
||||
mybackend_sources = [
|
||||
"csrc/storage_backends/mybackend/pybind.cpp",
|
||||
"csrc/storage_backends/mybackend/connector.cpp",
|
||||
]
|
||||
|
||||
# Add to ext_modules list:
|
||||
cpp_extension.CppExtension(
|
||||
"lmcache.lmcache_mybackend",
|
||||
sources=mybackend_sources,
|
||||
include_dirs=[
|
||||
"csrc/storage_backends",
|
||||
"csrc/storage_backends/mybackend",
|
||||
],
|
||||
extra_compile_args={"cxx": ["-O3", "-std=c++17"]},
|
||||
),
|
||||
|
||||
Then rebuild:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install -e .
|
||||
|
||||
|
||||
Step 4: Python Client (Non-MP Mode)
|
||||
------------------------------------
|
||||
|
||||
Inherit from ``ConnectorClientBase`` which provides asyncio event loop integration,
|
||||
future management, and both sync and async methods.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/v1/storage_backend/native_clients/mybackend_client.py
|
||||
from .connector_client_base import ConnectorClientBase
|
||||
from lmcache.lmcache_mybackend import LMCacheMyBackendClient
|
||||
|
||||
class MyBackendClient(ConnectorClientBase[LMCacheMyBackendClient]):
|
||||
def __init__(self, host: str, port: int,
|
||||
num_workers: int, loop=None):
|
||||
native = LMCacheMyBackendClient(host, port, num_workers)
|
||||
super().__init__(native, loop)
|
||||
|
||||
This gives you ``batch_get``, ``batch_set``, ``batch_exists`` (async), and their
|
||||
synchronous variants, all with automatic eventfd-driven completion handling.
|
||||
|
||||
**Reference:** ``lmcache/v1/storage_backend/native_clients/resp_client.py``
|
||||
|
||||
|
||||
Step 5: L2 Adapter (MP Mode)
|
||||
-----------------------------
|
||||
|
||||
To use your connector as an L2 adapter in MP mode, create a single Python module that
|
||||
defines the config class, factory function, and self-registers both. The
|
||||
``NativeConnectorL2Adapter`` bridge handles all the complexity (eventfd demuxing,
|
||||
key serialization, locking).
|
||||
|
||||
Create a new file in the L2 adapters package:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# lmcache/v1/distributed/l2_adapters/mybackend_l2_adapter.py
|
||||
from __future__ import annotations
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from lmcache.v1.distributed.internal_api import L1MemoryDesc
|
||||
|
||||
from lmcache.v1.distributed.l2_adapters.base import (
|
||||
L2AdapterInterface,
|
||||
)
|
||||
from lmcache.v1.distributed.l2_adapters.config import (
|
||||
L2AdapterConfigBase,
|
||||
register_l2_adapter_type,
|
||||
)
|
||||
from lmcache.v1.distributed.l2_adapters.factory import (
|
||||
register_l2_adapter_factory,
|
||||
)
|
||||
|
||||
|
||||
class MyBackendL2AdapterConfig(L2AdapterConfigBase):
|
||||
def __init__(self, host: str, port: int,
|
||||
num_workers: int = 8,
|
||||
max_capacity_gb: float = 0):
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.num_workers = num_workers
|
||||
self.max_capacity_gb = max_capacity_gb
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> "MyBackendL2AdapterConfig":
|
||||
host = d.get("host")
|
||||
if not isinstance(host, str) or not host:
|
||||
raise ValueError("host must be a non-empty string")
|
||||
port = d.get("port")
|
||||
if not isinstance(port, int) or port <= 0:
|
||||
raise ValueError("port must be a positive integer")
|
||||
num_workers = d.get("num_workers", 8)
|
||||
max_capacity_gb = d.get("max_capacity_gb", 0)
|
||||
return cls(host=host, port=port,
|
||||
num_workers=num_workers,
|
||||
max_capacity_gb=max_capacity_gb)
|
||||
|
||||
@classmethod
|
||||
def help(cls) -> str:
|
||||
return (
|
||||
"MyBackend L2 adapter config fields:\n"
|
||||
"- host (str): server hostname (required)\n"
|
||||
"- port (int): server port (required)\n"
|
||||
"- num_workers (int): worker threads (default 8)"
|
||||
)
|
||||
|
||||
|
||||
def _create_mybackend_l2_adapter(
|
||||
config: L2AdapterConfigBase,
|
||||
l1_memory_desc: "Optional[L1MemoryDesc]" = None,
|
||||
) -> L2AdapterInterface:
|
||||
from lmcache.lmcache_mybackend import LMCacheMyBackendClient
|
||||
from lmcache.v1.distributed.l2_adapters \
|
||||
.native_connector_l2_adapter import (
|
||||
NativeConnectorL2Adapter,
|
||||
)
|
||||
|
||||
assert isinstance(config, MyBackendL2AdapterConfig)
|
||||
native_client = LMCacheMyBackendClient(
|
||||
config.host, config.port, config.num_workers
|
||||
)
|
||||
return NativeConnectorL2Adapter(
|
||||
native_client,
|
||||
max_capacity_gb=config.max_capacity_gb,
|
||||
)
|
||||
|
||||
|
||||
# Self-register -- runs automatically when the module
|
||||
# is imported by the L2 adapter auto-discovery mechanism
|
||||
register_l2_adapter_type("mybackend", MyBackendL2AdapterConfig)
|
||||
register_l2_adapter_factory("mybackend", _create_mybackend_l2_adapter)
|
||||
|
||||
.. note::
|
||||
The L2 adapter package uses ``pkgutil.iter_modules`` to auto-discover all modules
|
||||
in ``lmcache/v1/distributed/l2_adapters/``. Simply creating the file above is
|
||||
sufficient -- no changes to ``__init__.py`` or any other existing file are needed.
|
||||
|
||||
**Usage from the command line:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache server \
|
||||
--l1-size-gb 100 --eviction-policy LRU \
|
||||
--l2-adapter '{"type": "mybackend", "host": "10.0.0.1", "port": 9000}'
|
||||
|
||||
|
||||
How NativeConnectorL2Adapter Bridges the Gap
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The C++ connector has 1 eventfd and mixed completions. MP mode's ``L2AdapterInterface``
|
||||
requires 3 separate eventfds and typed results. The bridge handles this transparently:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 25 45
|
||||
|
||||
* - L2 Adapter Method
|
||||
- Native Call
|
||||
- Extra Logic
|
||||
* - ``submit_store_task(keys, objs)``
|
||||
- ``submit_batch_set``
|
||||
- ObjectKey to string, MemoryObj to memoryview
|
||||
* - ``submit_lookup_and_lock_task(keys)``
|
||||
- ``submit_batch_exists``
|
||||
- + client-side lock refcount
|
||||
* - ``submit_load_task(keys, objs)``
|
||||
- ``submit_batch_get``
|
||||
- ObjectKey to string, MemoryObj to memoryview
|
||||
* - ``submit_unlock(keys)``
|
||||
- *(none)*
|
||||
- client-side lock decrement
|
||||
* - ``pop_completed_store_tasks()``
|
||||
- via ``drain_completions``
|
||||
- demux by op type
|
||||
* - ``query_lookup_and_lock_result()``
|
||||
- via ``drain_completions``
|
||||
- exists results to Bitmap, apply locks
|
||||
* - ``query_load_result()``
|
||||
- via ``drain_completions``
|
||||
- ok/fail to Bitmap
|
||||
|
||||
A background demux thread polls the native eventfd, calls ``drain_completions()``,
|
||||
looks up each ``future_id`` to determine its operation type, routes the result to
|
||||
the correct completion dict, and signals the corresponding Python eventfd.
|
||||
|
||||
|
||||
Third-Party Native Connector Plugins (``native_plugin``)
|
||||
---------------------------------------------------------
|
||||
|
||||
.. _native-plugin-overview:
|
||||
|
||||
The steps above describe how to add a native connector **inside** the LMCache source tree.
|
||||
If you want to ship a connector as a **separate, pip-installable package** (e.g. a proprietary
|
||||
storage backend), use the ``native_plugin`` L2 adapter type instead. It dynamically loads your
|
||||
connector at runtime -- no LMCache source modifications required.
|
||||
|
||||
How It Works
|
||||
~~~~~~~~~~~~
|
||||
|
||||
The ``native_plugin`` adapter type loads a third-party **connector object** (pybind-wrapped C++
|
||||
or pure Python) and wraps it with the built-in ``NativeConnectorL2Adapter`` bridge. This means
|
||||
you only need to implement the 6 connector methods -- the Python-side demux/lock bridging logic
|
||||
is reused from LMCache.
|
||||
|
||||
.. list-table:: ``plugin`` vs ``native_plugin``
|
||||
:header-rows: 1
|
||||
:widths: 25 35 40
|
||||
|
||||
* - Aspect
|
||||
- ``plugin``
|
||||
- ``native_plugin``
|
||||
* - What is loaded
|
||||
- A full ``L2AdapterInterface`` subclass
|
||||
- A **connector** object (lower level)
|
||||
* - Bridging logic
|
||||
- Provided by the plugin itself
|
||||
- Reused from ``NativeConnectorL2Adapter``
|
||||
* - Third-party effort
|
||||
- Must implement all abstract methods + 3 eventfds
|
||||
- Only 6 connector methods
|
||||
|
||||
Required Connector Interface
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The dynamically loaded connector instance must expose the following methods (identical to the
|
||||
pybind ``LMCACHE_BIND_CONNECTOR_METHODS`` contract):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class NativeConnectorProtocol:
|
||||
def event_fd(self) -> int: ...
|
||||
def submit_batch_get(
|
||||
self,
|
||||
keys: list[str],
|
||||
memoryviews: list[memoryview],
|
||||
) -> int: ...
|
||||
def submit_batch_set(
|
||||
self,
|
||||
keys: list[str],
|
||||
memoryviews: list[memoryview],
|
||||
) -> int: ...
|
||||
def submit_batch_exists(
|
||||
self,
|
||||
keys: list[str],
|
||||
) -> int: ...
|
||||
def submit_batch_delete(
|
||||
self,
|
||||
keys: list[str],
|
||||
) -> int: ...
|
||||
def drain_completions(
|
||||
self,
|
||||
) -> list[tuple[int, bool, str, list[bool] | None]]: ...
|
||||
def close(self) -> None: ...
|
||||
|
||||
The factory validates the first 6 methods at creation time and raises ``TypeError`` if
|
||||
any are missing. ``submit_batch_delete`` is **optional** -- if absent, the adapter's
|
||||
``delete()`` method will be a no-op (eviction will not remove keys from the backend).
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"type": "native_plugin",
|
||||
"module_path": "my_ext_package",
|
||||
"class_name": "MyConnectorClient",
|
||||
"adapter_params": {
|
||||
"host": "localhost",
|
||||
"port": 1234
|
||||
}
|
||||
}
|
||||
|
||||
.. list-table:: ``NativePluginL2AdapterConfig`` fields
|
||||
:header-rows: 1
|
||||
:widths: 20 10 10 60
|
||||
|
||||
* - Field
|
||||
- Type
|
||||
- Required
|
||||
- Description
|
||||
* - ``module_path``
|
||||
- ``str``
|
||||
- yes
|
||||
- Dotted Python import path of the module containing the connector class.
|
||||
* - ``class_name``
|
||||
- ``str``
|
||||
- yes
|
||||
- Name of the connector class inside ``module_path``.
|
||||
* - ``adapter_params``
|
||||
- ``dict``
|
||||
- no
|
||||
- Forwarded as ``**kwargs`` to the connector class constructor.
|
||||
* - ``max_capacity_gb``
|
||||
- ``float``
|
||||
- no
|
||||
- Maximum L2 storage capacity in GB for client-side usage tracking. Required for L2 eviction. Default 0 (disabled).
|
||||
|
||||
Loading Flow
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
CLI / config JSON
|
||||
|
|
||||
v
|
||||
NativePluginL2AdapterConfig.from_dict(d)
|
||||
|
|
||||
v
|
||||
_create_native_plugin_l2_adapter(config, ...)
|
||||
|
|
||||
+-- importlib.import_module(config.module_path)
|
||||
+-- getattr(module, config.class_name)
|
||||
+-- connector_cls(**config.adapter_params)
|
||||
+-- validate 6 required methods
|
||||
+-- NativeConnectorL2Adapter(native_client)
|
||||
|
|
||||
v
|
||||
L2AdapterInterface instance (ready for use)
|
||||
|
||||
Step-by-Step: Building an External Native Connector Plugin
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. **Create a Python package** with a C++ pybind11 extension that inherits from
|
||||
``ConnectorBase<T>`` (same base class as built-in connectors).
|
||||
|
||||
Project layout:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
my_ext_connector/
|
||||
+-- csrc/
|
||||
| +-- connector.h # C++ connector class
|
||||
| +-- connector.cpp # C++ implementation
|
||||
| +-- pybind.cpp # pybind11 bindings
|
||||
+-- src/
|
||||
| +-- my_ext_connector/
|
||||
| +-- __init__.py # re-export the factory class
|
||||
| +-- connector.py # Python factory wrapper
|
||||
+-- pyproject.toml
|
||||
+-- setup.py # build C++ extension
|
||||
|
||||
2. **Implement the C++ connector** inheriting from ``ConnectorBase<T>`` and override
|
||||
the 4 required methods (``create_connection``, ``do_single_get``, ``do_single_set``,
|
||||
``do_single_exists``) and optionally ``do_single_delete`` for eviction support.
|
||||
|
||||
3. **Create pybind11 bindings** using the ``LMCACHE_BIND_CONNECTOR_METHODS`` macro:
|
||||
|
||||
.. code-block:: cpp
|
||||
|
||||
#include <pybind11/pybind11.h>
|
||||
#include "connector_pybind_utils.h"
|
||||
#include "connector.h"
|
||||
|
||||
namespace py = pybind11;
|
||||
|
||||
PYBIND11_MODULE(_native, m) {
|
||||
py::class_<MyFSConnector>(m, "MyFSConnector")
|
||||
.def(py::init<std::string, int>(),
|
||||
py::arg("base_path"),
|
||||
py::arg("num_workers"))
|
||||
LMCACHE_BIND_CONNECTOR_METHODS(MyFSConnector);
|
||||
}
|
||||
|
||||
4. **Write a Python factory class** that selects the backend and returns the native
|
||||
connector instance:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from my_ext_connector._native import MyFSConnector
|
||||
|
||||
class MyConnectorClient:
|
||||
def __new__(
|
||||
cls,
|
||||
base_path: str = "/tmp/my_ext",
|
||||
num_workers: int = 2,
|
||||
):
|
||||
return MyFSConnector(base_path, num_workers)
|
||||
|
||||
5. **Build and install** the package:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd my_ext_connector
|
||||
pip install -e .
|
||||
|
||||
6. **Configure LMCache** to use it:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
--l2-adapter '{
|
||||
"type": "native_plugin",
|
||||
"module_path": "my_ext_connector",
|
||||
"class_name": "MyConnectorClient",
|
||||
"adapter_params": {
|
||||
"base_path": "/tmp/my_ext",
|
||||
"num_workers": 2
|
||||
}
|
||||
}'
|
||||
|
||||
Reference Implementation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
See ``examples/lmc_external_native_connector/`` for a complete, pip-installable example
|
||||
connector plugin that demonstrates:
|
||||
|
||||
- C++ pybind11-wrapped connectors inheriting from ``ConnectorBase<T>`` (same as built-in
|
||||
Redis/FS).
|
||||
- Two backends: filesystem (``ExampleFSConnector``) and in-memory
|
||||
(``ExampleMemoryConnector``), both in C++.
|
||||
- A thin Python factory class (``ExampleNativeConnector``) that selects the backend via a
|
||||
``"backend"`` parameter.
|
||||
- Worker thread pool with eventfd notification (inherited from ``ConnectorBase``).
|
||||
- Build via ``pip install -e .`` using pybind11 + setuptools.
|
||||
|
||||
|
||||
Checklist
|
||||
---------
|
||||
|
||||
Use this checklist when adding a new native connector:
|
||||
|
||||
1. C++ connector inheriting ``ConnectorBase<T>`` with 4 required + 1 optional (``do_single_delete``) method overrides
|
||||
2. Pybind module using ``LMCACHE_BIND_CONNECTOR_METHODS``
|
||||
3. ``setup.py`` entry for the new ``CppExtension``
|
||||
4. Python client inheriting ``ConnectorClientBase`` (non-MP mode)
|
||||
5. L2 adapter module with config class + factory self-registration (MP mode)
|
||||
6. Unit tests (see ``tests/v1/distributed/test_native_connector_l2_adapter.py``)
|
||||
7. Rebuild with ``pip install -e .`` and verify both modes work
|
||||
|
||||
For **external** native connector plugins (``native_plugin``):
|
||||
|
||||
1. Separate pip-installable package with C++ pybind11 extension
|
||||
2. Connector class exposing the 6 required methods (+ optional ``submit_batch_delete`` for eviction)
|
||||
3. Python factory class for backend selection
|
||||
4. ``pip install -e .`` and configure via ``--l2-adapter`` JSON
|
||||
5. Unit tests (see ``examples/lmc_external_native_connector/tests/``)
|
||||
|
||||
|
||||
Built-in Aerospike backend (optional)
|
||||
-------------------------------------
|
||||
|
||||
LMCache ships an optional in-tree Aerospike native connector (same
|
||||
``ConnectorBase`` harness as Redis). It is compiled only when
|
||||
``BUILD_AEROSPIKE=1`` or ``AEROSPIKE_INCLUDE_DIR`` is set during
|
||||
``pip install -e .``.
|
||||
|
||||
**Build:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
See ``.github/workflows/aerospike_integration.yml`` for installing the C client into ``.deps/``
|
||||
source .deps/aerospike-client-c.env
|
||||
BUILD_AEROSPIKE=1 pip install -e .
|
||||
|
||||
The ``aerospike-client-c.env`` file simply points the build at the C client
|
||||
headers and shared libraries you unpacked into ``.deps/``. Adjust the paths to
|
||||
match where the client was installed. Example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# .deps/aerospike-client-c.env
|
||||
export AEROSPIKE_INCLUDE_DIR="${PWD}/.deps/aerospike-install/usr/include"
|
||||
export AEROSPIKE_LIBRARY_DIR="${PWD}/.deps/aerospike-install/usr/lib"
|
||||
# Needed at build and runtime so the loader can find libaerospike (and
|
||||
# libyaml if you built it locally):
|
||||
export LD_LIBRARY_PATH="${AEROSPIKE_LIBRARY_DIR}:${PWD}/.deps/libyaml-install/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}"
|
||||
|
||||
Setting ``AEROSPIKE_INCLUDE_DIR`` is enough to enable the extension, so
|
||||
``BUILD_AEROSPIKE=1`` is optional once the env file is sourced. Multiple include
|
||||
or library directories can be passed as ``;``-separated lists.
|
||||
|
||||
**MP mode:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
--l2-adapter '{"type": "aerospike", "hosts": "127.0.0.1:3000", "namespace": "lmcache", "set_name": "kv_chunks", "num_workers": 8}'
|
||||
|
||||
Implementation: ``csrc/storage_backends/aerospike/``,
|
||||
``lmcache/v1/distributed/l2_adapters/aerospike_l2_adapter.py``.
|
||||
|
||||
|
||||
Additional Resources
|
||||
--------------------
|
||||
|
||||
- Framework source: ``csrc/storage_backends/``
|
||||
- ``ConnectorBase`` template: ``csrc/storage_backends/connector_base.h``
|
||||
- ``IStorageConnector`` interface: ``csrc/storage_backends/connector_interface.h``
|
||||
- Pybind utilities: ``csrc/storage_backends/connector_pybind_utils.h``
|
||||
- Redis reference implementation: ``csrc/storage_backends/redis/``
|
||||
- Aerospike implementation (optional): ``csrc/storage_backends/aerospike/``
|
||||
- Architecture README: ``csrc/storage_backends/README.md``
|
||||
- External native connector example:
|
||||
``examples/lmc_external_native_connector/``
|
||||
- Native plugin adapter source:
|
||||
``lmcache/v1/distributed/l2_adapters/native_connector_l2_adapter.py``
|
||||
- Design document:
|
||||
``lmcache/v1/distributed/l2_adapters/design_docs/plugin.md``
|
||||
- RESP backend user guide: :doc:`RESP (Native Redis/Valkey) <../../kv_cache/storage_backends/resp>`
|
||||
@@ -0,0 +1,294 @@
|
||||
Remote Storage Plugins
|
||||
========================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
LMCache supports built-in remote storage connectors for Redis, InfiniStore,
|
||||
MooncakeStore, S3, Hugging Face Buckets, and more.
|
||||
The remote storage plugin system provides the ability to add custom storage connectors through dynamic loading. This enables extending remote storage capabilities without modifying core code.
|
||||
|
||||
.. note::
|
||||
|
||||
The ``remote_url`` configuration is **deprecated** and will be removed in a future release.
|
||||
Please use ``remote_storage_plugins`` instead.
|
||||
|
||||
Connector Definition Requirements
|
||||
---------------------------------
|
||||
A custom remote storage connector requires two classes:
|
||||
|
||||
1. **ConnectorAdapter**: Handles URL scheme matching and connector instantiation
|
||||
|
||||
- Inherit from ``ConnectorAdapter``
|
||||
- Set the URL scheme in the constructor (e.g., ``mystore://``)
|
||||
- Implement the ``create_connector`` method
|
||||
|
||||
2. **RemoteConnector**: Implements the actual storage operations
|
||||
|
||||
- Inherit from ``RemoteConnector``
|
||||
- Implement all abstract methods: ``exists``, ``exists_sync``, ``get``, ``put``, ``list``, ``close``
|
||||
|
||||
.. note::
|
||||
|
||||
The ``ConnectorAdapter`` constructor receives no arguments from LMCache. The scheme should be set by calling the parent constructor with the scheme string.
|
||||
|
||||
The ``create_connector`` method receives a ``ConnectorContext`` object containing the URL, event loop, local CPU backend, config, metadata, and ``plugin_name``.
|
||||
|
||||
Plugin Naming Convention
|
||||
-------------------------
|
||||
Plugin names follow the format ``{type}`` or ``{type}.{instance}``:
|
||||
|
||||
- ``{type}`` — a single instance of that connector type (e.g. ``fs``, ``mooncakestore``)
|
||||
- ``{type}.{instance}`` — a named instance, allowing **multiple instances of the same type** (e.g. ``fs.primary``, ``fs.backup``)
|
||||
|
||||
The framework extracts the *type* portion (everything before the first ``.``) to locate the matching ``ConnectorAdapter``. The full plugin name is used as the configuration key prefix.
|
||||
|
||||
Using Built-in Connectors via Plugins
|
||||
-------------------------------------
|
||||
Built-in connectors (``fs``, ``mooncakestore``, ``hfbucket``, etc.) can be used
|
||||
directly via ``remote_storage_plugins`` without specifying ``module_path`` or
|
||||
``class_name``. Their configuration is placed under ``extra_config``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 64
|
||||
local_cpu: False
|
||||
max_local_cpu_size: 5
|
||||
remote_storage_plugins: ["fs"]
|
||||
extra_config:
|
||||
remote_storage_plugin.fs.base_path: /tmp/lmcache
|
||||
|
||||
Multiple instances of the same connector type:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
remote_storage_plugins: ["fs.primary", "fs.backup"]
|
||||
extra_config:
|
||||
remote_storage_plugin.fs.primary.base_path: /data/cache1
|
||||
remote_storage_plugin.fs.backup.base_path: /data/cache2
|
||||
|
||||
Mixing different connector types:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
remote_storage_plugins: ["fs.local", "mooncakestore"]
|
||||
extra_config:
|
||||
remote_storage_plugin.fs.local.base_path: /data/cache
|
||||
remote_storage_plugin.mooncakestore.master_server_address: "localhost:50051"
|
||||
|
||||
Built-in Hugging Face Buckets example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
remote_storage_plugins: ["hfbucket"]
|
||||
extra_config:
|
||||
remote_storage_plugin.hfbucket.bucket_handle: hf://buckets/my-org/lmcache-kv/prod
|
||||
remote_storage_plugin.hfbucket.token_env: HF_TOKEN
|
||||
|
||||
How to Integrate Custom Remote Storage with LMCache
|
||||
---------------------------------------------------
|
||||
1. Install your connector package in the LMCache environment
|
||||
2. Add ``remote_storage_plugins`` and its related ``module_path`` and ``class_name`` to the ``extra_config`` section of LMCache configuration as follows:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 64
|
||||
local_cpu: False
|
||||
max_local_cpu_size: 5
|
||||
remote_storage_plugins: ["mystore"]
|
||||
extra_config:
|
||||
remote_storage_plugin.mystore.module_path: <module_path>
|
||||
remote_storage_plugin.mystore.class_name: <adapter_class_name>
|
||||
|
||||
An example configuration for a custom remote storage connector:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 64
|
||||
local_cpu: False
|
||||
max_local_cpu_size: 5
|
||||
remote_storage_plugins: ["mystore"]
|
||||
extra_config:
|
||||
remote_storage_plugin.mystore.module_path: my_package.my_connector
|
||||
remote_storage_plugin.mystore.class_name: MyStoreConnectorAdapter
|
||||
|
||||
Multiple instances of a custom connector:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
remote_storage_plugins: ["mystore.region_a", "mystore.region_b"]
|
||||
extra_config:
|
||||
remote_storage_plugin.mystore.region_a.module_path: my_package.my_connector
|
||||
remote_storage_plugin.mystore.region_a.class_name: MyStoreConnectorAdapter
|
||||
remote_storage_plugin.mystore.region_b.module_path: my_package.my_connector
|
||||
remote_storage_plugin.mystore.region_b.class_name: MyStoreConnectorAdapter
|
||||
|
||||
.. note::
|
||||
|
||||
- ``remote_url`` is **deprecated**; use ``remote_storage_plugins`` instead
|
||||
- ``remote_storage_plugin.<plugin_name>`` uses the full plugin name (including instance suffix) as the key prefix
|
||||
- Multiple remote storage plugins can be loaded simultaneously
|
||||
- Built-in connectors do not require ``module_path`` / ``class_name``
|
||||
|
||||
ConnectorAdapter Implementation
|
||||
-------------------------------
|
||||
The ``ConnectorAdapter`` class is responsible for:
|
||||
|
||||
- Defining the URL scheme it handles
|
||||
- Creating the appropriate ``RemoteConnector`` instance
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from lmcache.v1.storage_backend.connector import (
|
||||
ConnectorAdapter,
|
||||
ConnectorContext,
|
||||
)
|
||||
from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
|
||||
|
||||
from lmcache.v1.storage_backend.connector import extract_plugin_type
|
||||
|
||||
PLUGIN_TYPE = "mystore"
|
||||
|
||||
class MyStoreConnectorAdapter(ConnectorAdapter):
|
||||
"""Adapter for MyStore remote storage."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
# Register the URL scheme this adapter handles
|
||||
super().__init__("mystore://")
|
||||
|
||||
def can_parse(self, url: str) -> bool:
|
||||
"""Match legacy URL or plugin://{type}[.{instance}] format."""
|
||||
if url.startswith(self.schema):
|
||||
return True
|
||||
if url.startswith("plugin://"):
|
||||
pname = url[len("plugin://"):]
|
||||
return extract_plugin_type(pname) == PLUGIN_TYPE
|
||||
return False
|
||||
|
||||
def create_connector(self, context: ConnectorContext) -> RemoteConnector:
|
||||
"""Create and return a MyStoreConnector instance."""
|
||||
# Access context properties as needed:
|
||||
# - context.url: the full remote URL
|
||||
# - context.loop: asyncio event loop
|
||||
# - context.config: LMCacheEngineConfig
|
||||
# - context.metadata: LMCacheMetadata
|
||||
# - context.plugin_name: plugin instance name
|
||||
# (e.g. "mystore", "mystore.region_a")
|
||||
return MyStoreConnector(
|
||||
context.config,
|
||||
context.metadata,
|
||||
plugin_name=context.plugin_name,
|
||||
)
|
||||
|
||||
RemoteConnector Implementation
|
||||
------------------------------
|
||||
The ``RemoteConnector`` class defines the interface for remote storage operations. Your implementation must provide the following abstract methods:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from typing import List, Optional
|
||||
from lmcache.utils import CacheEngineKey
|
||||
from lmcache.v1.config import LMCacheEngineConfig
|
||||
from lmcache.v1.metadata import LMCacheMetadata
|
||||
from lmcache.v1.memory_management import MemoryObj
|
||||
from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
|
||||
|
||||
class MyStoreConnector(RemoteConnector):
|
||||
"""Custom connector for MyStore remote storage."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: LMCacheEngineConfig,
|
||||
metadata: Optional[LMCacheMetadata]
|
||||
):
|
||||
super().__init__(config, metadata)
|
||||
# Initialize your connection here
|
||||
|
||||
async def exists(self, key: CacheEngineKey) -> bool:
|
||||
"""Check if a key exists in the remote store."""
|
||||
raise NotImplementedError
|
||||
|
||||
def exists_sync(self, key: CacheEngineKey) -> bool:
|
||||
"""Synchronous version of exists."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def get(self, key: CacheEngineKey) -> Optional[MemoryObj]:
|
||||
"""Retrieve a memory object by key. Return None if not found."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def put(self, key: CacheEngineKey, memory_obj: MemoryObj):
|
||||
"""Store a memory object with the given key."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def list(self) -> List[str]:
|
||||
"""List all keys in the remote store."""
|
||||
raise NotImplementedError
|
||||
|
||||
async def close(self):
|
||||
"""Close the connection to the remote store."""
|
||||
raise NotImplementedError
|
||||
|
||||
Optional Methods
|
||||
----------------
|
||||
The ``RemoteConnector`` base class also provides optional methods that can be overridden for enhanced functionality:
|
||||
|
||||
- ``support_ping()`` / ``ping()``: Health check support
|
||||
- ``support_batched_get()`` / ``batched_get()``: Batch retrieval operations
|
||||
- ``support_batched_put()`` / ``batched_put()``: Batch storage operations
|
||||
- ``support_batched_contains()`` / ``batched_contains()``: Batch existence checks
|
||||
- ``remove_sync()``: Synchronous key removal
|
||||
|
||||
Implementation Example
|
||||
----------------------
|
||||
A complete remote storage connector implementation would include both the adapter and connector classes in a single module or package. Here's a minimal working example structure:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
my_connector_package/
|
||||
├── __init__.py
|
||||
├── adapter.py # Contains MyStoreConnectorAdapter
|
||||
└── connector.py # Contains MyStoreConnector
|
||||
|
||||
The adapter module (``adapter.py``):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from lmcache.v1.storage_backend.connector import (
|
||||
ConnectorAdapter,
|
||||
ConnectorContext,
|
||||
extract_plugin_type,
|
||||
)
|
||||
from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
|
||||
from .connector import MyStoreConnector
|
||||
|
||||
PLUGIN_TYPE = "mystore"
|
||||
|
||||
class MyStoreConnectorAdapter(ConnectorAdapter):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("mystore://")
|
||||
|
||||
def can_parse(self, url: str) -> bool:
|
||||
if url.startswith(self.schema):
|
||||
return True
|
||||
if url.startswith("plugin://"):
|
||||
pname = url[len("plugin://"):]
|
||||
return extract_plugin_type(pname) == PLUGIN_TYPE
|
||||
return False
|
||||
|
||||
def create_connector(self, context: ConnectorContext) -> RemoteConnector:
|
||||
return MyStoreConnector(
|
||||
context.config,
|
||||
context.metadata,
|
||||
plugin_name=context.plugin_name,
|
||||
)
|
||||
|
||||
Configuration would then reference the adapter:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
remote_storage_plugins: ["mystore"]
|
||||
extra_config:
|
||||
remote_storage_plugin.mystore.module_path: my_connector_package.adapter
|
||||
remote_storage_plugin.mystore.class_name: MyStoreConnectorAdapter
|
||||
@@ -0,0 +1,89 @@
|
||||
Runtime Plugins
|
||||
===============
|
||||
|
||||
The LMCache runtime plugin system provides the ability to extend functionality by running custom scripts alongside LMCache processes. Plugins can be written in Python and Bash for now, and are managed by the ``RuntimePluginLauncher`` class.
|
||||
|
||||
Key Use Cases
|
||||
-------------
|
||||
- Start metric reporters for centralized monitoring
|
||||
- Implement log reporters for log collection systems
|
||||
- Report process-level metrics to alerting systems
|
||||
- Implement health checks and service discovery
|
||||
- Custom cache management operations
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
Runtime plugins are configured through environment variables and configuration files:
|
||||
|
||||
Environment Variables:
|
||||
- ``LMCACHE_RUNTIME_PLUGIN_ROLE``: Process role (e.g., ``SCHEDULER``, ``WORKER``)
|
||||
- ``LMCACHE_RUNTIME_PLUGIN_CONFIG``: JSON string containing plugin configuration
|
||||
- ``LMCACHE_RUNTIME_PLUGIN_WORKER_ID``: Current worker ID
|
||||
- ``LMCACHE_RUNTIME_PLUGIN_WORKER_COUNT``: Total worker count in cluster
|
||||
|
||||
Configuration File (``lmcache.yaml``):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
runtime_plugin_locations:
|
||||
- "/path/to/plugins"
|
||||
|
||||
extra_config:
|
||||
custom_setting: value
|
||||
|
||||
Runtime Plugin Naming Convention
|
||||
--------------------------------
|
||||
Plugin filenames determine execution targets:
|
||||
|
||||
Role-Specific Plugins:
|
||||
- Format: ``<ROLE>[_<WORKER_ID>][_<DESCRIPTION>].<EXTENSION>``
|
||||
- Examples:
|
||||
|
||||
- ``scheduler_foo_plugin.py``: Runs only on ``SCHEDULER``
|
||||
- ``worker_0_test.sh``: Runs only on worker ID 0
|
||||
- ``all_plugin.sh``: Runs on all workers
|
||||
|
||||
Notes:
|
||||
- Role names are case-insensitive
|
||||
- Worker ID must be numeric when specified
|
||||
- To target a specific worker ID, the filename must have at least three parts separated by underscores (e.g., `worker_<ID>_<DESCRIPTION>.ext`). A file named `worker_<DESCRIPTION>.ext` will run on all workers.
|
||||
|
||||
Execution Model
|
||||
---------------
|
||||
1. **Interpreter Detection**:
|
||||
- Uses shebang line (e.g., ``#!/opt/venv/bin/python``)
|
||||
- Fallback interpreters:
|
||||
|
||||
- ``.py`` → ``python``
|
||||
- ``.sh`` → ``bash``
|
||||
|
||||
2. **Output Handling**:
|
||||
- Stdout/stderr captured continuously
|
||||
- Logged with plugin name prefix
|
||||
|
||||
3. **Process Management**:
|
||||
- Launched as subprocesses
|
||||
- Terminated when parent process exits
|
||||
|
||||
Example Runtime Plugins
|
||||
-----------------------
|
||||
Python Plugin (``scheduler_foo_plugin.py``):
|
||||
|
||||
.. literalinclude:: ../../../../examples/plugins/scheduler_foo_plugin.py
|
||||
:language: python
|
||||
:linenos:
|
||||
|
||||
Bash Plugin (``all_plugin.sh``):
|
||||
|
||||
.. literalinclude:: ../../../../examples/plugins/all_plugin.sh
|
||||
:language: bash
|
||||
:linenos:
|
||||
|
||||
Best Practices
|
||||
--------------
|
||||
1. Keep runtime plugins lightweight and efficient
|
||||
2. Use descriptive naming conventions
|
||||
3. Implement graceful error handling
|
||||
4. Include shebang for portability
|
||||
5. Validate configuration inputs
|
||||
6. Add timeout mechanisms for long operations
|
||||
@@ -0,0 +1,248 @@
|
||||
Storage Plugins
|
||||
===============
|
||||
|
||||
LMCache supports out of the box storage backends like Mooncake, S3 and NIXL.
|
||||
The LMCache storage plugin system provides the ability to add custom storage backends through dynamic loading or plug and play capability. In other words, extending cache storage capabilities without modifying core code.
|
||||
|
||||
Backend Definition Requirements
|
||||
-------------------------------
|
||||
1. Inherit from ``StoragePluginInterface``
|
||||
2. Implement all the abstract methods of the parent interface of ``StoragePluginInterface``- ``StorageBackendInterface``
|
||||
3. Package as an installable Python module
|
||||
|
||||
.. note::
|
||||
|
||||
The interface constructor is the instantiation contract that the LMCache loading system will use when loading custom storage backends.
|
||||
If you wish to implement a constructor, it should have the same parameter signature and call the interface constructor.
|
||||
|
||||
How to Integrate the Backend with LMCache
|
||||
-----------------------------------------
|
||||
1. Install your backend package in the LMCache environment
|
||||
2. Add ``storage_plugins`` and its related ``module_path`` and ``class_name`` to ``extra_config`` section of LMCache configuration as follows:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 64
|
||||
local_cpu: False
|
||||
max_local_cpu_size: 5
|
||||
storage_plugins: <backend_name>
|
||||
extra_config:
|
||||
storage_plugin.<backend_name>.module_path: <module_path>
|
||||
storage_plugin.<backend_name>.class_name: <class_name>
|
||||
|
||||
An example configuration for a logging backend is as follows:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 64
|
||||
local_cpu: False
|
||||
max_local_cpu_size: 5
|
||||
storage_plugins: "log_backend"
|
||||
extra_config:
|
||||
storage_plugin.log_backend.module_path: lmc_external_log_backend.lmc_external_log_backend
|
||||
storage_plugin.log_backend.class_name: ExternalLogBackend
|
||||
|
||||
.. note::
|
||||
|
||||
- Storage backends are initialized in order during LMCache startup - earlier backends have higher priority during cache lookups
|
||||
- ``storage_plugin.<backend_name>`` distinguishes the different dynamic loaded backends
|
||||
|
||||
Backend Implementation Example
|
||||
------------------------------
|
||||
A sample custom backend implementation can be viewed at https://github.com/opendataio/lmc_external_log_backend/
|
||||
|
||||
|
||||
MP-Mode L2 Adapter Plugins (``plugin``)
|
||||
----------------------------------------
|
||||
|
||||
.. _plugin-l2-adapter-overview:
|
||||
|
||||
The storage plugin system described above applies to **non-MP mode** (single-process). For
|
||||
**MP mode** (multiprocess), LMCache provides the ``plugin`` L2 adapter type, which dynamically
|
||||
loads a third-party ``L2AdapterInterface`` implementation at runtime.
|
||||
|
||||
Overview
|
||||
~~~~~~~~
|
||||
|
||||
The ``plugin`` adapter type lets you ship a full L2 adapter as a **separate, pip-installable
|
||||
package**. At startup, LMCache imports your module, instantiates your adapter class, and uses it
|
||||
just like a built-in adapter -- no LMCache source modifications required.
|
||||
|
||||
.. note::
|
||||
|
||||
If your storage backend is a native C++ connector (pybind-wrapped), consider using the
|
||||
``native_plugin`` type instead (see :doc:`native_connectors`). It reuses the built-in
|
||||
bridging logic so you only need to implement 6 connector methods rather than the full
|
||||
``L2AdapterInterface``.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"type": "plugin",
|
||||
"module_path": "my_plugin.adapter",
|
||||
"class_name": "MyL2Adapter",
|
||||
"adapter_params": {
|
||||
"host": "localhost",
|
||||
"capacity": 1000
|
||||
}
|
||||
}
|
||||
|
||||
.. list-table:: ``PluginL2AdapterConfig`` fields
|
||||
:header-rows: 1
|
||||
:widths: 22 10 10 58
|
||||
|
||||
* - Field
|
||||
- Type
|
||||
- Required
|
||||
- Description
|
||||
* - ``module_path``
|
||||
- ``str``
|
||||
- yes
|
||||
- Dotted Python import path of the module containing the adapter class.
|
||||
* - ``class_name``
|
||||
- ``str``
|
||||
- yes
|
||||
- Name of the class inside ``module_path`` that implements ``L2AdapterInterface``.
|
||||
* - ``adapter_params``
|
||||
- ``dict``
|
||||
- no
|
||||
- Forwarded to the adapter class constructor (as a typed config or raw dict).
|
||||
* - ``config_class_name``
|
||||
- ``str``
|
||||
- no
|
||||
- Explicit config class name; when omitted the factory auto-discovers it.
|
||||
|
||||
Config Class Auto-Discovery
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The factory automatically resolves the adapter's config class using the following priority
|
||||
chain (first match wins):
|
||||
|
||||
1. **Explicit** ``config_class_name`` field in JSON config.
|
||||
2. **Convention**: adapter class name + ``"Config"`` suffix (e.g. ``MyL2Adapter`` looks for
|
||||
``MyL2AdapterConfig``).
|
||||
3. **Attribute**: ``config_class_name`` attribute on the adapter class itself.
|
||||
4. **Fallback**: no config class found -- pass raw ``adapter_params`` dict.
|
||||
|
||||
Each candidate is looked up in the loaded module and validated as an ``L2AdapterConfigBase``
|
||||
subclass. This means most plugins that follow the naming convention will **automatically**
|
||||
receive a typed config instance without any extra configuration.
|
||||
|
||||
Plugin Contract
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
A plugin adapter class **must**:
|
||||
|
||||
1. Subclass ``L2AdapterInterface`` from ``lmcache.v1.distributed.l2_adapters.base``.
|
||||
2. Implement all abstract methods: ``submit_store_task``, ``pop_completed_store_tasks``,
|
||||
``submit_lookup_and_lock_task``, ``query_lookup_and_lock_result``, ``submit_unlock``,
|
||||
``submit_load_task``, ``query_load_result``, ``close``, and all three event-fd getters.
|
||||
3. Provide **three distinct event fds** (store / lookup / load). The controllers build
|
||||
``fd -> adapter`` maps; duplicates will misroute events.
|
||||
4. Be **thread-safe**: ``StoreController`` and ``PrefetchController`` call adapter methods
|
||||
from different threads concurrently.
|
||||
5. Accept ``**kwargs`` in ``__init__`` to stay forward-compatible.
|
||||
|
||||
A plugin adapter class **should**:
|
||||
|
||||
1. Create its own asyncio event loop and background thread if it needs async I/O.
|
||||
2. Use ``create_event_notifier()`` from ``lmcache.v1.platform`` for the three event fds
|
||||
(cross-platform: ``os.eventfd`` on Linux, ``os.pipe`` fallback elsewhere).
|
||||
3. Clean up all resources (event fds, threads, connections) in ``close()``.
|
||||
|
||||
Loading Flow
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
CLI / config JSON
|
||||
|
|
||||
v
|
||||
PluginL2AdapterConfig.from_dict(d)
|
||||
| validates module_path, class_name, adapter_params
|
||||
|
|
||||
v
|
||||
_create_plugin_adapter(config, ...)
|
||||
|
|
||||
+-- importlib.import_module(config.module_path)
|
||||
+-- getattr(module, config.class_name)
|
||||
+-- issubclass check against L2AdapterInterface
|
||||
|
|
||||
+-- _resolve_config_class(module, config, adapter_cls)
|
||||
| +-- 1. config.config_class_name (explicit)
|
||||
| +-- 2. class_name + "Config" (convention)
|
||||
| +-- 3. adapter_cls.config_class_name (attribute)
|
||||
| +-- 4. None (fall back to raw dict)
|
||||
|
|
||||
+-- [if config class found]
|
||||
| +-- adapter_cls(cfg_cls.from_dict(adapter_params))
|
||||
|
|
||||
+-- [otherwise]
|
||||
+-- adapter_cls(adapter_params)
|
||||
|
|
||||
v
|
||||
L2AdapterInterface instance (ready for use)
|
||||
|
||||
Minimal Example
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# my_plugin/adapter.py
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
from lmcache.native_storage_ops import Bitmap
|
||||
from lmcache.v1.distributed.l2_adapters.base import (
|
||||
L2AdapterInterface,
|
||||
L2TaskId,
|
||||
)
|
||||
from lmcache.v1.platform import create_event_notifier
|
||||
|
||||
|
||||
class MyL2Adapter(L2AdapterInterface):
|
||||
def __init__(self, params, **_kw):
|
||||
self._store_efd = create_event_notifier()
|
||||
self._lookup_efd = create_event_notifier()
|
||||
self._load_efd = create_event_notifier()
|
||||
# ... set up connection, background thread, etc.
|
||||
|
||||
# implement all abstract methods ...
|
||||
|
||||
def close(self) -> None:
|
||||
self._store_efd.close()
|
||||
self._lookup_efd.close()
|
||||
self._load_efd.close()
|
||||
|
||||
Launch via CLI:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
--l2-adapter '{
|
||||
"type": "plugin",
|
||||
"module_path": "my_plugin.adapter",
|
||||
"class_name": "MyL2Adapter",
|
||||
"adapter_params": {"host": "localhost"}
|
||||
}'
|
||||
|
||||
Reference Implementation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
See ``examples/lmc_external_l2_adapter/`` for a complete, pip-installable example plugin
|
||||
(``InMemoryL2Adapter``) that demonstrates:
|
||||
|
||||
- FIFO eviction with configurable capacity.
|
||||
- Simulated bandwidth delay for realistic testing.
|
||||
- Background asyncio event loop with proper shutdown.
|
||||
- Full test suite covering store, lookup, load, batch operations, and eviction behavior.
|
||||
|
||||
Additional Resources
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Plugin adapter source: ``lmcache/v1/distributed/l2_adapters/plugin_l2_adapter.py``
|
||||
- Native plugin adapter: ``lmcache/v1/distributed/l2_adapters/native_connector_l2_adapter.py``
|
||||
- Design document: ``lmcache/v1/distributed/l2_adapters/design_docs/plugin.md``
|
||||
- L2 adapter base interface: ``lmcache/v1/distributed/l2_adapters/base.py``
|
||||
@@ -0,0 +1,13 @@
|
||||
Developer Guide
|
||||
===============
|
||||
|
||||
How to contribute to LMCache and extend it with new storage backends, CLI
|
||||
commands, and HTTP endpoints.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
contributing
|
||||
extending_lmcache/native_connectors
|
||||
cli
|
||||
extending_http_api
|
||||
@@ -0,0 +1,48 @@
|
||||
Integration
|
||||
===========
|
||||
|
||||
LMCache acts as a caching middleware that sits between LLM inference engines and storage systems, enabling:
|
||||
|
||||
- **Transparent Cache Reuse**: Automatic detection and reuse of previously computed KV caches
|
||||
- **Performance Optimization**: 3×–10× reduction in time-to-first-token (TTFT) for multi-round conversation and RAG.
|
||||
- **Resource Efficiency**: Significant GPU cycle savings through cache reuse
|
||||
- **Scalable Architecture**: Support for single-node and multi-node deployments
|
||||
|
||||
Supported Engines
|
||||
-----------------
|
||||
|
||||
LMCache currently supports integration with:
|
||||
|
||||
- **vLLM**
|
||||
- **SGLang**
|
||||
- **TRT-LLM** (coming soon)
|
||||
|
||||
Integration with vLLM
|
||||
---------------------
|
||||
|
||||
When LMCache is integrated with vLLM, the inference pipeline is augmented to lookup and inject cached KV chunks for any reused input content. The sequence diagram below shows how a prompt request flows through vLLM with LMCache:
|
||||
|
||||
.. mermaid::
|
||||
|
||||
sequenceDiagram
|
||||
participant User as User Application
|
||||
participant VLLM as vLLM Engine (LMCache Connector)
|
||||
participant Cache as LMCache Storage (CPU/Disk)
|
||||
User->>VLLM: Prompt request (LLM query)
|
||||
VLLM->>Cache: **Lookup** KV cache for prompt tokens
|
||||
alt Cache hit for some tokens
|
||||
Cache-->>VLLM: Return cached KV chunk(s)
|
||||
VLLM->>VLLM: Inject KV into model's cache (skip recomputation)
|
||||
else Cache miss (no KV for tokens)
|
||||
Cache-->>VLLM: No cached data (proceed normally)
|
||||
end
|
||||
VLLM->>VLLM: **Generate** remaining tokens (compute new KV)
|
||||
VLLM-->>User: Respond with completion
|
||||
Note right of VLLM: *LLM output is returned without waiting for cache storage.*
|
||||
VLLM->>Cache: **Store** new KV chunk(s) (async background put)
|
||||
|
||||
**Cache Lookup**: Upon receiving a new prompt, vLLM (via the LMCache connector) computes identifiers (e.g. hashes of token sequences) and queries LMCache for matching KV cache chunks. If a cache hit occurs, LMCache retrieves the KV chunk (potentially from CPU or disk) and returns it to vLLM. vLLM then injects these KV tensors into the model's attention cache instead of recomputing them from scratch.
|
||||
|
||||
**Inference with Cache**: vLLM proceeds with inference. For any parts of the prompt where no cache was available (cache miss), the model computes new KV values as usual. Cached segments (e.g. previously seen text) are skipped in computation, significantly reducing the time-to-first-token (TTFT) and saving GPU cycles.
|
||||
|
||||
**Async Cache Write**: After processing the prompt, any newly generated KV cache chunks (corresponding to this prompt's content) are handed off to LMCache for storage. This put operation is done asynchronously (in the background) so it doesn't delay the response. The response is returned to the user promptly, and LMCache's background tasks will offload the new KV data to CPU, disk, or other backends for future reuse.
|
||||
@@ -0,0 +1,144 @@
|
||||
Basic Check Tool
|
||||
================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
The LMCache Basic Check Tool is a testing and validation utility that helps you verify your LMCache installation, configuration, and functionality. It provides multiple testing modes to validate different components of the LMCache system.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The basic check tool (``lmcache.v1.basic_check``) is designed to:
|
||||
|
||||
* Test remote backend connectivity and functionality
|
||||
* Validate storage manager operations
|
||||
* Generate test keys for performance testing
|
||||
* Verify configuration settings
|
||||
* Provide diagnostic information for troubleshooting
|
||||
|
||||
Available Check Modes
|
||||
---------------------
|
||||
|
||||
The tool supports several check modes, each targeting specific functionality:
|
||||
|
||||
test_remote
|
||||
~~~~~~~~~~~
|
||||
|
||||
Tests the remote backend functionality including:
|
||||
|
||||
* Connection establishment to remote backends (fs, etc.)
|
||||
* put/get operations with data integrity validation
|
||||
* put/get/exists operations with performance reports
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode test_remote
|
||||
|
||||
test_storage_manager
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tests the storage manager operations including:
|
||||
|
||||
* Configuration validation
|
||||
* batched_put/get operations with data integrity validation
|
||||
* batched_put/get/contains operations with performance reports
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode test_storage_manager
|
||||
|
||||
gen (Key Generation)
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Generates test keys for performance testing and benchmarking:
|
||||
|
||||
* Configurable number of keys and concurrency levels
|
||||
* Memory-efficient batch processing
|
||||
* Progress tracking and performance metrics
|
||||
* Offset support for distributed testing
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode gen --num-keys 1000 --concurrency 16
|
||||
|
||||
Command Line Interface
|
||||
----------------------
|
||||
|
||||
Basic Usage
|
||||
~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode <MODE> [OPTIONS]
|
||||
|
||||
List Available Modes
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode list
|
||||
|
||||
Command Line Options
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. option:: --mode MODE
|
||||
|
||||
**Required.** Operation mode to run. Use ``list`` to see available modes.
|
||||
|
||||
.. option:: --model MODEL
|
||||
|
||||
Model name for testing, just a part of key of persist kv-cache. Default: ``/lmcache_test_model/``
|
||||
|
||||
.. option:: --num-keys NUM
|
||||
|
||||
Number of keys to generate (gen mode only). Default: 100
|
||||
|
||||
.. option:: --concurrency NUM
|
||||
|
||||
Concurrency level for operations (gen mode only). Default: 16
|
||||
|
||||
.. option:: --offset NUM
|
||||
|
||||
Offset for key generation (gen mode only). Default: 0
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The basic check tool uses your existing LMCache configuration. You can specify configuration in several ways:
|
||||
|
||||
Environment Variable
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export LMCACHE_CONFIG_PATH=/path/to/config.yaml
|
||||
python -m lmcache.v1.basic_check --mode test_remote
|
||||
|
||||
Example Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here's an example configuration optimized for basic checks:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Basic cache settings
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
max_local_cpu_size: 1.0 # 1GB for basic checks
|
||||
|
||||
# Remote backend (optional)
|
||||
remote_url: "file:///tmp/lmcache_basic_check"
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
The ``examples/basic_check/`` directory contains comprehensive examples:
|
||||
@@ -0,0 +1,7 @@
|
||||
Usage Data Module
|
||||
=================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
usage_stats_collection
|
||||
@@ -0,0 +1,155 @@
|
||||
.. _usage-stats-collection:
|
||||
|
||||
Usage Stats Collection
|
||||
======================
|
||||
|
||||
LMCache collects anonymous usage data by default to help the engineering team understand real-world workloads, prioritize optimizations, and improve reliability. All collected data is aggregated and contains no sensitive user information.
|
||||
|
||||
A sanitized subset of the aggregated data may be publicly released for the community’s benefit (for example, see a daily usage report `here <https://github.com/Hanchenli/OSS_Growth_Toolkit/tree/main/usage_tracker/report>`_).
|
||||
|
||||
What data is collected?
|
||||
-----------------------
|
||||
|
||||
Usage stats are implemented in the ``lmcache/usage_telemetry/`` package. The
|
||||
one-shot messages sent at startup depend on how LMCache runs.
|
||||
|
||||
When LMCache runs **inside the serving engine** (the single-process
|
||||
integrations for vLLM, SGLang, and TensorRT-LLM), engine startup sends:
|
||||
|
||||
- **EnvMessage**
|
||||
Captures environment details such as cloud provider, CPU info, total memory, architecture, GPU count/type, and execution source.
|
||||
|
||||
- **EngineMessage**
|
||||
Records engine configuration and metadata, including cache settings (chunk size, local device, cache limits), remote backend parameters, blending settings, model name, world size, and KV-cache dtype/shape.
|
||||
|
||||
- **MetadataMessage**
|
||||
Reports execution metadata: the timestamp when the run started and total duration in seconds.
|
||||
|
||||
When LMCache runs as a **standalone multiprocess (MP) cache server**
|
||||
(``lmcache server``), server startup sends:
|
||||
|
||||
- **EnvMessage**
|
||||
Same environment snapshot as above, taken on the cache-server host.
|
||||
|
||||
- **MPServerMessage**
|
||||
Records the server configuration: LMCache version, chunk size, hash
|
||||
algorithm, engine type, transfer mode, worker pool sizes, whether P2P is
|
||||
enabled, L1 size and medium (DRAM / GDS / DRAM+DevDAX), eviction policy,
|
||||
the configured L2 adapter and serde types, and the L2 store/prefetch
|
||||
policies.
|
||||
|
||||
The MP server's ``--instance-id`` is **never** sent: it can be
|
||||
operator-chosen and therefore identifying. Model names are not known at
|
||||
server startup (vLLM instances register later) and are not part of this
|
||||
message.
|
||||
|
||||
In addition to the one-shot messages above, a continuous reporter periodically
|
||||
sends interval counters (**ContinuousContextMessage**: tokens stored/hit and
|
||||
stored KV bytes in the interval) and a cache-lifespan histogram
|
||||
(**CacheLifespanMessage**). The flush interval is controlled by
|
||||
``LMCACHE_USAGE_TRACK_INTERVAL`` (seconds, default 600).
|
||||
|
||||
Every payload carries four common fields:
|
||||
|
||||
- ``session_id`` -- a random UUID minted once per process, joining the
|
||||
one-shot context with the continuous messages of the same run.
|
||||
- ``machine_id`` -- a random UUID persisted at
|
||||
``~/.config/lmcache/machine_id``, grouping sessions from the same machine.
|
||||
It is never derived from hardware identifiers (MAC address, hostname).
|
||||
- ``schema_version`` -- the version of the message schema.
|
||||
- ``deployment_mode`` -- ``single_process`` (LMCache inside the serving
|
||||
engine) or ``mp_server`` (standalone MP cache server).
|
||||
|
||||
These messages are serialized to JSON and POSTed to the LMCache usage server.
|
||||
|
||||
Example JSON payload
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"message_type": "EnvMessage",
|
||||
"provider": "GCP",
|
||||
"num_cpu": 24,
|
||||
"cpu_type": "Intel(R) Xeon(R) CPU @ 2.20GHz",
|
||||
"cpu_family_model_stepping": "6,85,7",
|
||||
"total_memory": 101261135872,
|
||||
"architecture": ["64bit", "ELF"],
|
||||
"platforms": "Linux-5.10.0-28-cloud-amd64-x86_64-with-glibc2.31",
|
||||
"gpu_count": 2,
|
||||
"gpu_type": "NVIDIA L4",
|
||||
"gpu_memory_per_device": 23580639232,
|
||||
"source": "DOCKER"
|
||||
}
|
||||
|
||||
Previewing collected data
|
||||
-------------------------
|
||||
|
||||
If you enable **local logging**, usage messages are appended to your specified log file. To inspect the most recent entries:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
tail ~/.config/lmcache/usage.log
|
||||
|
||||
Configuration & Opt-out
|
||||
-----------------------
|
||||
|
||||
By default, usage tracking is **enabled**. Any one of the following opt-outs
|
||||
disables all usage stats collection:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# LMCache-specific opt-out
|
||||
export LMCACHE_TRACK_USAGE=false
|
||||
|
||||
# The cross-tool "do not track" convention (1/true/yes)
|
||||
export DO_NOT_TRACK=1
|
||||
|
||||
When tracking is disabled, ``InitializeUsageContext`` will return ``None`` and
|
||||
no data will be sent or logged, and no state files (such as ``machine_id``)
|
||||
will be created.
|
||||
|
||||
Reference
|
||||
~~~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 34 14 52
|
||||
|
||||
* - Environment variable / file
|
||||
- Default
|
||||
- Effect
|
||||
* - ``LMCACHE_TRACK_USAGE``
|
||||
- unset
|
||||
- Set to ``false`` to disable all usage stats collection.
|
||||
* - ``DO_NOT_TRACK``
|
||||
- unset
|
||||
- Set to ``1``/``true``/``yes`` to disable collection (cross-tool
|
||||
convention).
|
||||
* - ``LMCACHE_USAGE_TRACK_URL``
|
||||
- ``http://stats.lmcache.ai:8080``
|
||||
- Override the stats server endpoint (e.g. for a private sink).
|
||||
* - ``LMCACHE_USAGE_TRACK_INTERVAL``
|
||||
- ``600``
|
||||
- Seconds between continuous-message flushes.
|
||||
* - ``~/.config/lmcache/machine_id``
|
||||
- created on first send
|
||||
- Holds the random anonymous machine UUID. Delete it to rotate the
|
||||
identifier.
|
||||
|
||||
Local logging
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
If you would like to log to a file in addition to (or instead of) sending data to the server, pass a local-log path when initializing:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from lmcache.usage_telemetry import InitializeUsageContext
|
||||
|
||||
usage_ctx = InitializeUsageContext(
|
||||
config=engine_config,
|
||||
metadata=engine_metadata,
|
||||
local_log="~/.config/lmcache/usage.log"
|
||||
)
|
||||
|
||||
Omitting the ``local_log`` argument (or passing ``None``) disables local file logging.
|
||||
Reference in New Issue
Block a user