chore: import upstream snapshot with attribution

docs/source/developer_guide/extending_lmcache/native_connectors.rst

@@ -0,0 +1,686 @@
+Adding Native Backends
+======================
+
+.. _native-connectors-overview:
+
+Overview
+--------
+
+Native connectors are high-performance C++ storage backends that integrate with LMCache
+through pybind11. They work in **both** LMCache operating modes:
+
+- **Non-MP mode** (single process): via ``ConnectorClientBase`` (asyncio integration)
+- **MP mode** (multiprocess): via ``NativeConnectorL2Adapter`` (L2 adapter interface)
+
+Write the connector once, get both modes for free.
+
+The framework lives in ``csrc/storage_backends/`` with the Redis RESP connector as the
+reference implementation.
+
+Architecture
+~~~~~~~~~~~~
+
+.. code-block:: text
+
+    Non-MP mode:
+      CacheEngine -> RemoteBackend -> ConnectorClientBase -> native client (C++)
+                                        (asyncio event loop)
+
+    MP mode:
+      StoreController / PrefetchController
+            |
+      NativeConnectorL2Adapter (Python bridge)
+        +-- 3 eventfds (store, lookup, load)
+        +-- completion demux thread
+        +-- ObjectKey <-> string serialization
+        +-- client-side lock tracking
+            |
+      native client (C++)
+        +-- 1 eventfd, worker threads, GIL-free I/O
+
+**Design principles:**
+
+1. **GIL release** at the pybind layer for true concurrency between native threads
+2. **Batching with tiling**: work for a batched request is split evenly among threads
+3. **eventfd-based completions**: the kernel wakes Python -- no polling
+4. **Non-blocking submission**: submission queue / completion queue architecture
+
+
+Step 1: C++ Connector
+---------------------
+
+Create your connector directory (e.g., ``csrc/storage_backends/mybackend/``) and
+inherit from ``ConnectorBase<YourConnectionType>``. You need to override 4 required methods
+(and optionally ``do_single_delete`` to support eviction).
+
+**connector.h:**
+
+.. code-block:: cpp
+
+    // csrc/storage_backends/mybackend/connector.h
+    #pragma once
+    #include "../connector_base.h"
+
+    namespace lmcache {
+    namespace connector {
+
+    // Per-thread connection state
+    struct MyConn {
+      int fd = -1;
+      // your connection fields
+    };
+
+    class MyConnector : public ConnectorBase<MyConn> {
+     public:
+      MyConnector(std::string host, int port, int num_workers)
+          : ConnectorBase(num_workers), host_(host), port_(port) {
+        start_workers();  // IMPORTANT: call at END of constructor
+      }
+
+     protected:
+      // 1. Create a connection (called once per worker thread)
+      MyConn create_connection() override {
+        MyConn conn;
+        // connect to server...
+        return conn;
+      }
+
+      // 2. GET: read value for key into buf
+      void do_single_get(MyConn& conn, const std::string& key,
+                         void* buf, size_t len,
+                         size_t chunk_size) override {
+        // send GET command, recv response into buf
+      }
+
+      // 3. SET: write data from buf under key
+      void do_single_set(MyConn& conn, const std::string& key,
+                         const void* buf, size_t len,
+                         size_t chunk_size) override {
+        // send SET command with data from buf
+      }
+
+      // 4. EXISTS: check if key exists
+      bool do_single_exists(MyConn& conn,
+                            const std::string& key) override {
+        // send EXISTS, return true/false
+      }
+
+      // 5. DELETE: remove key (optional, has default no-op)
+      bool do_single_delete(MyConn& conn,
+                            const std::string& key) override {
+        // send DELETE, return true if deleted, false if not found
+      }
+
+      // Optional: clean shutdown
+      void shutdown_connections() override {
+        // close sockets, free resources
+      }
+
+     private:
+      std::string host_;
+      int port_;
+    };
+
+    }  // namespace connector
+    }  // namespace lmcache
+
+**What ConnectorBase gives you for free:**
+
+- Worker thread pool with per-thread connections
+- Submission queue (lock-free enqueue) and completion queue
+- Automatic tiling: batch operations are split across workers
+- eventfd signaling on completion (kernel wakes Python)
+- Graceful shutdown (stop flag, drain, join)
+
+.. important::
+   Always call ``start_workers()`` at the **end** of your derived constructor,
+   after all member variables are initialized. Worker threads call
+   ``create_connection()`` immediately, so the object must be fully constructed.
+
+**Reference:** ``csrc/storage_backends/redis/connector.h`` and ``connector.cpp``
+
+
+Step 2: Pybind Module
+---------------------
+
+Use the ``LMCACHE_BIND_CONNECTOR_METHODS`` macro, which binds all 7 methods
+(``event_fd``, ``submit_batch_get/set/exists/delete``, ``drain_completions``, ``close``)
+with proper GIL release and Python buffer protocol handling.
+
+.. code-block:: cpp
+
+    // csrc/storage_backends/mybackend/pybind.cpp
+    #include <pybind11/pybind11.h>
+    #include "../connector_pybind_utils.h"
+    #include "connector.h"
+
+    namespace py = pybind11;
+
+    PYBIND11_MODULE(lmcache_mybackend, m) {
+      py::class_<lmcache::connector::MyConnector>(m, "LMCacheMyBackendClient")
+          .def(py::init<std::string, int, int>(),
+               py::arg("host"), py::arg("port"),
+               py::arg("num_workers"))
+          LMCACHE_BIND_CONNECTOR_METHODS(
+              lmcache::connector::MyConnector);
+    }
+
+The pybind utilities automatically:
+
+- Extract buffer pointers from Python ``memoryview`` objects under the GIL
+- Release the GIL before calling into C++
+- Convert C++ ``Completion`` structs to Python tuples ``(future_id, ok, error, result_bools)``
+
+**Reference:** ``csrc/storage_backends/redis/pybind.cpp``
+
+
+Step 3: Build System
+--------------------
+
+Register your C++ sources in ``setup.py`` alongside the existing Redis extension:
+
+.. code-block:: python
+
+    # In cuda_extension() and rocm_extension():
+    mybackend_sources = [
+        "csrc/storage_backends/mybackend/pybind.cpp",
+        "csrc/storage_backends/mybackend/connector.cpp",
+    ]
+
+    # Add to ext_modules list:
+    cpp_extension.CppExtension(
+        "lmcache.lmcache_mybackend",
+        sources=mybackend_sources,
+        include_dirs=[
+            "csrc/storage_backends",
+            "csrc/storage_backends/mybackend",
+        ],
+        extra_compile_args={"cxx": ["-O3", "-std=c++17"]},
+    ),
+
+Then rebuild:
+
+.. code-block:: bash
+
+    pip install -e .
+
+
+Step 4: Python Client (Non-MP Mode)
+------------------------------------
+
+Inherit from ``ConnectorClientBase`` which provides asyncio event loop integration,
+future management, and both sync and async methods.
+
+.. code-block:: python
+
+    # lmcache/v1/storage_backend/native_clients/mybackend_client.py
+    from .connector_client_base import ConnectorClientBase
+    from lmcache.lmcache_mybackend import LMCacheMyBackendClient
+
+    class MyBackendClient(ConnectorClientBase[LMCacheMyBackendClient]):
+        def __init__(self, host: str, port: int,
+                     num_workers: int, loop=None):
+            native = LMCacheMyBackendClient(host, port, num_workers)
+            super().__init__(native, loop)
+
+This gives you ``batch_get``, ``batch_set``, ``batch_exists`` (async), and their
+synchronous variants, all with automatic eventfd-driven completion handling.
+
+**Reference:** ``lmcache/v1/storage_backend/native_clients/resp_client.py``
+
+
+Step 5: L2 Adapter (MP Mode)
+-----------------------------
+
+To use your connector as an L2 adapter in MP mode, create a single Python module that
+defines the config class, factory function, and self-registers both. The
+``NativeConnectorL2Adapter`` bridge handles all the complexity (eventfd demuxing,
+key serialization, locking).
+
+Create a new file in the L2 adapters package:
+
+.. code-block:: python
+
+    # lmcache/v1/distributed/l2_adapters/mybackend_l2_adapter.py
+    from __future__ import annotations
+    from typing import TYPE_CHECKING, Optional
+
+    if TYPE_CHECKING:
+        from lmcache.v1.distributed.internal_api import L1MemoryDesc
+
+    from lmcache.v1.distributed.l2_adapters.base import (
+        L2AdapterInterface,
+    )
+    from lmcache.v1.distributed.l2_adapters.config import (
+        L2AdapterConfigBase,
+        register_l2_adapter_type,
+    )
+    from lmcache.v1.distributed.l2_adapters.factory import (
+        register_l2_adapter_factory,
+    )
+
+
+    class MyBackendL2AdapterConfig(L2AdapterConfigBase):
+        def __init__(self, host: str, port: int,
+                     num_workers: int = 8,
+                     max_capacity_gb: float = 0):
+            self.host = host
+            self.port = port
+            self.num_workers = num_workers
+            self.max_capacity_gb = max_capacity_gb
+
+        @classmethod
+        def from_dict(cls, d: dict) -> "MyBackendL2AdapterConfig":
+            host = d.get("host")
+            if not isinstance(host, str) or not host:
+                raise ValueError("host must be a non-empty string")
+            port = d.get("port")
+            if not isinstance(port, int) or port <= 0:
+                raise ValueError("port must be a positive integer")
+            num_workers = d.get("num_workers", 8)
+            max_capacity_gb = d.get("max_capacity_gb", 0)
+            return cls(host=host, port=port,
+                       num_workers=num_workers,
+                       max_capacity_gb=max_capacity_gb)
+
+        @classmethod
+        def help(cls) -> str:
+            return (
+                "MyBackend L2 adapter config fields:\n"
+                "- host (str): server hostname (required)\n"
+                "- port (int): server port (required)\n"
+                "- num_workers (int): worker threads (default 8)"
+            )
+
+
+    def _create_mybackend_l2_adapter(
+        config: L2AdapterConfigBase,
+        l1_memory_desc: "Optional[L1MemoryDesc]" = None,
+    ) -> L2AdapterInterface:
+        from lmcache.lmcache_mybackend import LMCacheMyBackendClient
+        from lmcache.v1.distributed.l2_adapters \
+            .native_connector_l2_adapter import (
+            NativeConnectorL2Adapter,
+        )
+
+        assert isinstance(config, MyBackendL2AdapterConfig)
+        native_client = LMCacheMyBackendClient(
+            config.host, config.port, config.num_workers
+        )
+        return NativeConnectorL2Adapter(
+            native_client,
+            max_capacity_gb=config.max_capacity_gb,
+        )
+
+
+    # Self-register -- runs automatically when the module
+    # is imported by the L2 adapter auto-discovery mechanism
+    register_l2_adapter_type("mybackend", MyBackendL2AdapterConfig)
+    register_l2_adapter_factory("mybackend", _create_mybackend_l2_adapter)
+
+.. note::
+   The L2 adapter package uses ``pkgutil.iter_modules`` to auto-discover all modules
+   in ``lmcache/v1/distributed/l2_adapters/``. Simply creating the file above is
+   sufficient -- no changes to ``__init__.py`` or any other existing file are needed.
+
+**Usage from the command line:**
+
+.. code-block:: bash
+
+    lmcache server \
+        --l1-size-gb 100 --eviction-policy LRU \
+        --l2-adapter '{"type": "mybackend", "host": "10.0.0.1", "port": 9000}'
+
+
+How NativeConnectorL2Adapter Bridges the Gap
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The C++ connector has 1 eventfd and mixed completions. MP mode's ``L2AdapterInterface``
+requires 3 separate eventfds and typed results. The bridge handles this transparently:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 25 45
+
+   * - L2 Adapter Method
+     - Native Call
+     - Extra Logic
+   * - ``submit_store_task(keys, objs)``
+     - ``submit_batch_set``
+     - ObjectKey to string, MemoryObj to memoryview
+   * - ``submit_lookup_and_lock_task(keys)``
+     - ``submit_batch_exists``
+     - + client-side lock refcount
+   * - ``submit_load_task(keys, objs)``
+     - ``submit_batch_get``
+     - ObjectKey to string, MemoryObj to memoryview
+   * - ``submit_unlock(keys)``
+     - *(none)*
+     - client-side lock decrement
+   * - ``pop_completed_store_tasks()``
+     - via ``drain_completions``
+     - demux by op type
+   * - ``query_lookup_and_lock_result()``
+     - via ``drain_completions``
+     - exists results to Bitmap, apply locks
+   * - ``query_load_result()``
+     - via ``drain_completions``
+     - ok/fail to Bitmap
+
+A background demux thread polls the native eventfd, calls ``drain_completions()``,
+looks up each ``future_id`` to determine its operation type, routes the result to
+the correct completion dict, and signals the corresponding Python eventfd.
+
+
+Third-Party Native Connector Plugins (``native_plugin``)
+---------------------------------------------------------
+
+.. _native-plugin-overview:
+
+The steps above describe how to add a native connector **inside** the LMCache source tree.
+If you want to ship a connector as a **separate, pip-installable package** (e.g. a proprietary
+storage backend), use the ``native_plugin`` L2 adapter type instead. It dynamically loads your
+connector at runtime -- no LMCache source modifications required.
+
+How It Works
+~~~~~~~~~~~~
+
+The ``native_plugin`` adapter type loads a third-party **connector object** (pybind-wrapped C++
+or pure Python) and wraps it with the built-in ``NativeConnectorL2Adapter`` bridge. This means
+you only need to implement the 6 connector methods -- the Python-side demux/lock bridging logic
+is reused from LMCache.
+
+.. list-table:: ``plugin`` vs ``native_plugin``
+   :header-rows: 1
+   :widths: 25 35 40
+
+   * - Aspect
+     - ``plugin``
+     - ``native_plugin``
+   * - What is loaded
+     - A full ``L2AdapterInterface`` subclass
+     - A **connector** object (lower level)
+   * - Bridging logic
+     - Provided by the plugin itself
+     - Reused from ``NativeConnectorL2Adapter``
+   * - Third-party effort
+     - Must implement all abstract methods + 3 eventfds
+     - Only 6 connector methods
+
+Required Connector Interface
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The dynamically loaded connector instance must expose the following methods (identical to the
+pybind ``LMCACHE_BIND_CONNECTOR_METHODS`` contract):
+
+.. code-block:: python
+
+    class NativeConnectorProtocol:
+        def event_fd(self) -> int: ...
+        def submit_batch_get(
+            self,
+            keys: list[str],
+            memoryviews: list[memoryview],
+        ) -> int: ...
+        def submit_batch_set(
+            self,
+            keys: list[str],
+            memoryviews: list[memoryview],
+        ) -> int: ...
+        def submit_batch_exists(
+            self,
+            keys: list[str],
+        ) -> int: ...
+        def submit_batch_delete(
+            self,
+            keys: list[str],
+        ) -> int: ...
+        def drain_completions(
+            self,
+        ) -> list[tuple[int, bool, str, list[bool] | None]]: ...
+        def close(self) -> None: ...
+
+The factory validates the first 6 methods at creation time and raises ``TypeError`` if
+any are missing. ``submit_batch_delete`` is **optional** -- if absent, the adapter's
+``delete()`` method will be a no-op (eviction will not remove keys from the backend).
+
+Configuration
+~~~~~~~~~~~~~
+
+.. code-block:: json
+
+    {
+      "type": "native_plugin",
+      "module_path": "my_ext_package",
+      "class_name": "MyConnectorClient",
+      "adapter_params": {
+        "host": "localhost",
+        "port": 1234
+      }
+    }
+
+.. list-table:: ``NativePluginL2AdapterConfig`` fields
+   :header-rows: 1
+   :widths: 20 10 10 60
+
+   * - Field
+     - Type
+     - Required
+     - Description
+   * - ``module_path``
+     - ``str``
+     - yes
+     - Dotted Python import path of the module containing the connector class.
+   * - ``class_name``
+     - ``str``
+     - yes
+     - Name of the connector class inside ``module_path``.
+   * - ``adapter_params``
+     - ``dict``
+     - no
+     - Forwarded as ``**kwargs`` to the connector class constructor.
+   * - ``max_capacity_gb``
+     - ``float``
+     - no
+     - Maximum L2 storage capacity in GB for client-side usage tracking. Required for L2 eviction. Default 0 (disabled).
+
+Loading Flow
+~~~~~~~~~~~~
+
+.. code-block:: text
+
+    CLI / config JSON
+      |
+      v
+    NativePluginL2AdapterConfig.from_dict(d)
+      |
+      v
+    _create_native_plugin_l2_adapter(config, ...)
+      |
+      +-- importlib.import_module(config.module_path)
+      +-- getattr(module, config.class_name)
+      +-- connector_cls(**config.adapter_params)
+      +-- validate 6 required methods
+      +-- NativeConnectorL2Adapter(native_client)
+              |
+              v
+      L2AdapterInterface instance (ready for use)
+
+Step-by-Step: Building an External Native Connector Plugin
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. **Create a Python package** with a C++ pybind11 extension that inherits from
+   ``ConnectorBase<T>`` (same base class as built-in connectors).
+
+   Project layout:
+
+   .. code-block:: text
+
+       my_ext_connector/
+       +-- csrc/
+       |   +-- connector.h      # C++ connector class
+       |   +-- connector.cpp    # C++ implementation
+       |   +-- pybind.cpp       # pybind11 bindings
+       +-- src/
+       |   +-- my_ext_connector/
+       |       +-- __init__.py   # re-export the factory class
+       |       +-- connector.py  # Python factory wrapper
+       +-- pyproject.toml
+       +-- setup.py              # build C++ extension
+
+2. **Implement the C++ connector** inheriting from ``ConnectorBase<T>`` and override
+   the 4 required methods (``create_connection``, ``do_single_get``, ``do_single_set``,
+   ``do_single_exists``) and optionally ``do_single_delete`` for eviction support.
+
+3. **Create pybind11 bindings** using the ``LMCACHE_BIND_CONNECTOR_METHODS`` macro:
+
+   .. code-block:: cpp
+
+       #include <pybind11/pybind11.h>
+       #include "connector_pybind_utils.h"
+       #include "connector.h"
+
+       namespace py = pybind11;
+
+       PYBIND11_MODULE(_native, m) {
+         py::class_<MyFSConnector>(m, "MyFSConnector")
+             .def(py::init<std::string, int>(),
+                  py::arg("base_path"),
+                  py::arg("num_workers"))
+             LMCACHE_BIND_CONNECTOR_METHODS(MyFSConnector);
+       }
+
+4. **Write a Python factory class** that selects the backend and returns the native
+   connector instance:
+
+   .. code-block:: python
+
+       from my_ext_connector._native import MyFSConnector
+
+       class MyConnectorClient:
+           def __new__(
+               cls,
+               base_path: str = "/tmp/my_ext",
+               num_workers: int = 2,
+           ):
+               return MyFSConnector(base_path, num_workers)
+
+5. **Build and install** the package:
+
+   .. code-block:: bash
+
+       cd my_ext_connector
+       pip install -e .
+
+6. **Configure LMCache** to use it:
+
+   .. code-block:: bash
+
+       --l2-adapter '{
+         "type": "native_plugin",
+         "module_path": "my_ext_connector",
+         "class_name": "MyConnectorClient",
+         "adapter_params": {
+           "base_path": "/tmp/my_ext",
+           "num_workers": 2
+         }
+       }'
+
+Reference Implementation
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See ``examples/lmc_external_native_connector/`` for a complete, pip-installable example
+connector plugin that demonstrates:
+
+- C++ pybind11-wrapped connectors inheriting from ``ConnectorBase<T>`` (same as built-in
+  Redis/FS).
+- Two backends: filesystem (``ExampleFSConnector``) and in-memory
+  (``ExampleMemoryConnector``), both in C++.
+- A thin Python factory class (``ExampleNativeConnector``) that selects the backend via a
+  ``"backend"`` parameter.
+- Worker thread pool with eventfd notification (inherited from ``ConnectorBase``).
+- Build via ``pip install -e .`` using pybind11 + setuptools.
+
+
+Checklist
+---------
+
+Use this checklist when adding a new native connector:
+
+1. C++ connector inheriting ``ConnectorBase<T>`` with 4 required + 1 optional (``do_single_delete``) method overrides
+2. Pybind module using ``LMCACHE_BIND_CONNECTOR_METHODS``
+3. ``setup.py`` entry for the new ``CppExtension``
+4. Python client inheriting ``ConnectorClientBase`` (non-MP mode)
+5. L2 adapter module with config class + factory self-registration (MP mode)
+6. Unit tests (see ``tests/v1/distributed/test_native_connector_l2_adapter.py``)
+7. Rebuild with ``pip install -e .`` and verify both modes work
+
+For **external** native connector plugins (``native_plugin``):
+
+1. Separate pip-installable package with C++ pybind11 extension
+2. Connector class exposing the 6 required methods (+ optional ``submit_batch_delete`` for eviction)
+3. Python factory class for backend selection
+4. ``pip install -e .`` and configure via ``--l2-adapter`` JSON
+5. Unit tests (see ``examples/lmc_external_native_connector/tests/``)
+
+
+Built-in Aerospike backend (optional)
+-------------------------------------
+
+LMCache ships an optional in-tree Aerospike native connector (same
+``ConnectorBase`` harness as Redis). It is compiled only when
+``BUILD_AEROSPIKE=1`` or ``AEROSPIKE_INCLUDE_DIR`` is set during
+``pip install -e .``.
+
+**Build:**
+
+.. code-block:: bash
+
+   See ``.github/workflows/aerospike_integration.yml`` for installing the C client into ``.deps/``
+   source .deps/aerospike-client-c.env
+   BUILD_AEROSPIKE=1 pip install -e .
+
+The ``aerospike-client-c.env`` file simply points the build at the C client
+headers and shared libraries you unpacked into ``.deps/``. Adjust the paths to
+match where the client was installed. Example:
+
+.. code-block:: bash
+
+   # .deps/aerospike-client-c.env
+   export AEROSPIKE_INCLUDE_DIR="${PWD}/.deps/aerospike-install/usr/include"
+   export AEROSPIKE_LIBRARY_DIR="${PWD}/.deps/aerospike-install/usr/lib"
+   # Needed at build and runtime so the loader can find libaerospike (and
+   # libyaml if you built it locally):
+   export LD_LIBRARY_PATH="${AEROSPIKE_LIBRARY_DIR}:${PWD}/.deps/libyaml-install/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}"
+
+Setting ``AEROSPIKE_INCLUDE_DIR`` is enough to enable the extension, so
+``BUILD_AEROSPIKE=1`` is optional once the env file is sourced. Multiple include
+or library directories can be passed as ``;``-separated lists.
+
+**MP mode:**
+
+.. code-block:: bash
+
+   --l2-adapter '{"type": "aerospike", "hosts": "127.0.0.1:3000", "namespace": "lmcache", "set_name": "kv_chunks", "num_workers": 8}'
+
+Implementation: ``csrc/storage_backends/aerospike/``,
+``lmcache/v1/distributed/l2_adapters/aerospike_l2_adapter.py``.
+
+
+Additional Resources
+--------------------
+
+- Framework source: ``csrc/storage_backends/``
+- ``ConnectorBase`` template: ``csrc/storage_backends/connector_base.h``
+- ``IStorageConnector`` interface: ``csrc/storage_backends/connector_interface.h``
+- Pybind utilities: ``csrc/storage_backends/connector_pybind_utils.h``
+- Redis reference implementation: ``csrc/storage_backends/redis/``
+- Aerospike implementation (optional): ``csrc/storage_backends/aerospike/``
+- Architecture README: ``csrc/storage_backends/README.md``
+- External native connector example:
+  ``examples/lmc_external_native_connector/``
+- Native plugin adapter source:
+  ``lmcache/v1/distributed/l2_adapters/native_connector_l2_adapter.py``
+- Design document:
+  ``lmcache/v1/distributed/l2_adapters/design_docs/plugin.md``
+- RESP backend user guide: :doc:`RESP (Native Redis/Valkey) <../../kv_cache/storage_backends/resp>`

docs/source/developer_guide/extending_lmcache/remote_storage_plugins.rst

@@ -0,0 +1,294 @@
+Remote Storage Plugins
+========================
+
+.. warning::
+
+   This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
+
+
+LMCache supports built-in remote storage connectors for Redis, InfiniStore,
+MooncakeStore, S3, Hugging Face Buckets, and more.
+The remote storage plugin system provides the ability to add custom storage connectors through dynamic loading. This enables extending remote storage capabilities without modifying core code.
+
+.. note::
+
+   The ``remote_url`` configuration is **deprecated** and will be removed in a future release.
+   Please use ``remote_storage_plugins`` instead.
+
+Connector Definition Requirements
+---------------------------------
+A custom remote storage connector requires two classes:
+
+1. **ConnectorAdapter**: Handles URL scheme matching and connector instantiation
+
+   - Inherit from ``ConnectorAdapter``
+   - Set the URL scheme in the constructor (e.g., ``mystore://``)
+   - Implement the ``create_connector`` method
+
+2. **RemoteConnector**: Implements the actual storage operations
+
+   - Inherit from ``RemoteConnector``
+   - Implement all abstract methods: ``exists``, ``exists_sync``, ``get``, ``put``, ``list``, ``close``
+
+.. note::
+
+   The ``ConnectorAdapter`` constructor receives no arguments from LMCache. The scheme should be set by calling the parent constructor with the scheme string.
+
+   The ``create_connector`` method receives a ``ConnectorContext`` object containing the URL, event loop, local CPU backend, config, metadata, and ``plugin_name``.
+
+Plugin Naming Convention
+-------------------------
+Plugin names follow the format ``{type}`` or ``{type}.{instance}``:
+
+- ``{type}`` — a single instance of that connector type (e.g. ``fs``, ``mooncakestore``)
+- ``{type}.{instance}`` — a named instance, allowing **multiple instances of the same type** (e.g. ``fs.primary``, ``fs.backup``)
+
+The framework extracts the *type* portion (everything before the first ``.``) to locate the matching ``ConnectorAdapter``. The full plugin name is used as the configuration key prefix.
+
+Using Built-in Connectors via Plugins
+-------------------------------------
+Built-in connectors (``fs``, ``mooncakestore``, ``hfbucket``, etc.) can be used
+directly via ``remote_storage_plugins`` without specifying ``module_path`` or
+``class_name``. Their configuration is placed under ``extra_config``:
+
+.. code-block:: yaml
+
+    chunk_size: 64
+    local_cpu: False
+    max_local_cpu_size: 5
+    remote_storage_plugins: ["fs"]
+    extra_config:
+      remote_storage_plugin.fs.base_path: /tmp/lmcache
+
+Multiple instances of the same connector type:
+
+.. code-block:: yaml
+
+    remote_storage_plugins: ["fs.primary", "fs.backup"]
+    extra_config:
+      remote_storage_plugin.fs.primary.base_path: /data/cache1
+      remote_storage_plugin.fs.backup.base_path: /data/cache2
+
+Mixing different connector types:
+
+.. code-block:: yaml
+
+    remote_storage_plugins: ["fs.local", "mooncakestore"]
+    extra_config:
+      remote_storage_plugin.fs.local.base_path: /data/cache
+      remote_storage_plugin.mooncakestore.master_server_address: "localhost:50051"
+
+Built-in Hugging Face Buckets example:
+
+.. code-block:: yaml
+
+    remote_storage_plugins: ["hfbucket"]
+    extra_config:
+      remote_storage_plugin.hfbucket.bucket_handle: hf://buckets/my-org/lmcache-kv/prod
+      remote_storage_plugin.hfbucket.token_env: HF_TOKEN
+
+How to Integrate Custom Remote Storage with LMCache
+---------------------------------------------------
+1. Install your connector package in the LMCache environment
+2. Add ``remote_storage_plugins`` and its related ``module_path`` and ``class_name`` to the ``extra_config`` section of LMCache configuration as follows:
+
+.. code-block:: yaml
+
+    chunk_size: 64
+    local_cpu: False
+    max_local_cpu_size: 5
+    remote_storage_plugins: ["mystore"]
+    extra_config:
+      remote_storage_plugin.mystore.module_path: <module_path>
+      remote_storage_plugin.mystore.class_name: <adapter_class_name>
+
+An example configuration for a custom remote storage connector:
+
+.. code-block:: yaml
+
+    chunk_size: 64
+    local_cpu: False
+    max_local_cpu_size: 5
+    remote_storage_plugins: ["mystore"]
+    extra_config:
+      remote_storage_plugin.mystore.module_path: my_package.my_connector
+      remote_storage_plugin.mystore.class_name: MyStoreConnectorAdapter
+
+Multiple instances of a custom connector:
+
+.. code-block:: yaml
+
+    remote_storage_plugins: ["mystore.region_a", "mystore.region_b"]
+    extra_config:
+      remote_storage_plugin.mystore.region_a.module_path: my_package.my_connector
+      remote_storage_plugin.mystore.region_a.class_name: MyStoreConnectorAdapter
+      remote_storage_plugin.mystore.region_b.module_path: my_package.my_connector
+      remote_storage_plugin.mystore.region_b.class_name: MyStoreConnectorAdapter
+
+.. note::
+
+   - ``remote_url`` is **deprecated**; use ``remote_storage_plugins`` instead
+   - ``remote_storage_plugin.<plugin_name>`` uses the full plugin name (including instance suffix) as the key prefix
+   - Multiple remote storage plugins can be loaded simultaneously
+   - Built-in connectors do not require ``module_path`` / ``class_name``
+
+ConnectorAdapter Implementation
+-------------------------------
+The ``ConnectorAdapter`` class is responsible for:
+
+- Defining the URL scheme it handles
+- Creating the appropriate ``RemoteConnector`` instance
+
+.. code-block:: python
+
+    from lmcache.v1.storage_backend.connector import (
+        ConnectorAdapter,
+        ConnectorContext,
+    )
+    from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
+
+    from lmcache.v1.storage_backend.connector import extract_plugin_type
+
+    PLUGIN_TYPE = "mystore"
+
+    class MyStoreConnectorAdapter(ConnectorAdapter):
+        """Adapter for MyStore remote storage."""
+
+        def __init__(self) -> None:
+            # Register the URL scheme this adapter handles
+            super().__init__("mystore://")
+
+        def can_parse(self, url: str) -> bool:
+            """Match legacy URL or plugin://{type}[.{instance}] format."""
+            if url.startswith(self.schema):
+                return True
+            if url.startswith("plugin://"):
+                pname = url[len("plugin://"):]
+                return extract_plugin_type(pname) == PLUGIN_TYPE
+            return False
+
+        def create_connector(self, context: ConnectorContext) -> RemoteConnector:
+            """Create and return a MyStoreConnector instance."""
+            # Access context properties as needed:
+            # - context.url: the full remote URL
+            # - context.loop: asyncio event loop
+            # - context.config: LMCacheEngineConfig
+            # - context.metadata: LMCacheMetadata
+            # - context.plugin_name: plugin instance name
+            #   (e.g. "mystore", "mystore.region_a")
+            return MyStoreConnector(
+                context.config,
+                context.metadata,
+                plugin_name=context.plugin_name,
+            )
+
+RemoteConnector Implementation
+------------------------------
+The ``RemoteConnector`` class defines the interface for remote storage operations. Your implementation must provide the following abstract methods:
+
+.. code-block:: python
+
+    from typing import List, Optional
+    from lmcache.utils import CacheEngineKey
+    from lmcache.v1.config import LMCacheEngineConfig
+    from lmcache.v1.metadata import LMCacheMetadata
+    from lmcache.v1.memory_management import MemoryObj
+    from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
+
+    class MyStoreConnector(RemoteConnector):
+        """Custom connector for MyStore remote storage."""
+
+        def __init__(
+            self,
+            config: LMCacheEngineConfig,
+            metadata: Optional[LMCacheMetadata]
+        ):
+            super().__init__(config, metadata)
+            # Initialize your connection here
+
+        async def exists(self, key: CacheEngineKey) -> bool:
+            """Check if a key exists in the remote store."""
+            raise NotImplementedError
+
+        def exists_sync(self, key: CacheEngineKey) -> bool:
+            """Synchronous version of exists."""
+            raise NotImplementedError
+
+        async def get(self, key: CacheEngineKey) -> Optional[MemoryObj]:
+            """Retrieve a memory object by key. Return None if not found."""
+            raise NotImplementedError
+
+        async def put(self, key: CacheEngineKey, memory_obj: MemoryObj):
+            """Store a memory object with the given key."""
+            raise NotImplementedError
+
+        async def list(self) -> List[str]:
+            """List all keys in the remote store."""
+            raise NotImplementedError
+
+        async def close(self):
+            """Close the connection to the remote store."""
+            raise NotImplementedError
+
+Optional Methods
+----------------
+The ``RemoteConnector`` base class also provides optional methods that can be overridden for enhanced functionality:
+
+- ``support_ping()`` / ``ping()``: Health check support
+- ``support_batched_get()`` / ``batched_get()``: Batch retrieval operations
+- ``support_batched_put()`` / ``batched_put()``: Batch storage operations
+- ``support_batched_contains()`` / ``batched_contains()``: Batch existence checks
+- ``remove_sync()``: Synchronous key removal
+
+Implementation Example
+----------------------
+A complete remote storage connector implementation would include both the adapter and connector classes in a single module or package. Here's a minimal working example structure:
+
+.. code-block:: text
+
+    my_connector_package/
+    ├── __init__.py
+    ├── adapter.py      # Contains MyStoreConnectorAdapter
+    └── connector.py    # Contains MyStoreConnector
+
+The adapter module (``adapter.py``):
+
+.. code-block:: python
+
+    from lmcache.v1.storage_backend.connector import (
+        ConnectorAdapter,
+        ConnectorContext,
+        extract_plugin_type,
+    )
+    from lmcache.v1.storage_backend.connector.base_connector import RemoteConnector
+    from .connector import MyStoreConnector
+
+    PLUGIN_TYPE = "mystore"
+
+    class MyStoreConnectorAdapter(ConnectorAdapter):
+        def __init__(self) -> None:
+            super().__init__("mystore://")
+
+        def can_parse(self, url: str) -> bool:
+            if url.startswith(self.schema):
+                return True
+            if url.startswith("plugin://"):
+                pname = url[len("plugin://"):]
+                return extract_plugin_type(pname) == PLUGIN_TYPE
+            return False
+
+        def create_connector(self, context: ConnectorContext) -> RemoteConnector:
+            return MyStoreConnector(
+                context.config,
+                context.metadata,
+                plugin_name=context.plugin_name,
+            )
+
+Configuration would then reference the adapter:
+
+.. code-block:: yaml
+
+    remote_storage_plugins: ["mystore"]
+    extra_config:
+      remote_storage_plugin.mystore.module_path: my_connector_package.adapter
+      remote_storage_plugin.mystore.class_name: MyStoreConnectorAdapter

docs/source/developer_guide/extending_lmcache/runtime_plugins.rst

@@ -0,0 +1,89 @@
+Runtime Plugins
+===============
+
+The LMCache runtime plugin system provides the ability to extend functionality by running custom scripts alongside LMCache processes. Plugins can be written in Python and Bash for now, and are managed by the ``RuntimePluginLauncher`` class.
+
+Key Use Cases
+-------------
+- Start metric reporters for centralized monitoring
+- Implement log reporters for log collection systems
+- Report process-level metrics to alerting systems
+- Implement health checks and service discovery
+- Custom cache management operations
+
+Configuration
+-------------
+Runtime plugins are configured through environment variables and configuration files:
+
+Environment Variables:
+- ``LMCACHE_RUNTIME_PLUGIN_ROLE``: Process role (e.g., ``SCHEDULER``, ``WORKER``)
+- ``LMCACHE_RUNTIME_PLUGIN_CONFIG``: JSON string containing plugin configuration
+- ``LMCACHE_RUNTIME_PLUGIN_WORKER_ID``: Current worker ID
+- ``LMCACHE_RUNTIME_PLUGIN_WORKER_COUNT``: Total worker count in cluster
+
+Configuration File (``lmcache.yaml``):
+
+.. code-block:: yaml
+
+   runtime_plugin_locations:
+     - "/path/to/plugins"
+
+   extra_config:
+     custom_setting: value
+
+Runtime Plugin Naming Convention
+--------------------------------
+Plugin filenames determine execution targets:
+
+Role-Specific Plugins:
+- Format: ``<ROLE>[_<WORKER_ID>][_<DESCRIPTION>].<EXTENSION>``
+- Examples:
+
+  - ``scheduler_foo_plugin.py``: Runs only on ``SCHEDULER``
+  - ``worker_0_test.sh``: Runs only on worker ID 0
+  - ``all_plugin.sh``: Runs on all workers
+
+Notes:
+- Role names are case-insensitive
+- Worker ID must be numeric when specified
+- To target a specific worker ID, the filename must have at least three parts separated by underscores (e.g., `worker_<ID>_<DESCRIPTION>.ext`). A file named `worker_<DESCRIPTION>.ext` will run on all workers.
+
+Execution Model
+---------------
+1. **Interpreter Detection**:
+   - Uses shebang line (e.g., ``#!/opt/venv/bin/python``)
+   - Fallback interpreters:
+
+     - ``.py`` → ``python``
+     - ``.sh`` → ``bash``
+
+2. **Output Handling**:
+   - Stdout/stderr captured continuously
+   - Logged with plugin name prefix
+
+3. **Process Management**:
+   - Launched as subprocesses
+   - Terminated when parent process exits
+
+Example Runtime Plugins
+-----------------------
+Python Plugin (``scheduler_foo_plugin.py``):
+
+.. literalinclude:: ../../../../examples/plugins/scheduler_foo_plugin.py
+   :language: python
+   :linenos:
+
+Bash Plugin (``all_plugin.sh``):
+
+.. literalinclude:: ../../../../examples/plugins/all_plugin.sh
+   :language: bash
+   :linenos:
+
+Best Practices
+--------------
+1. Keep runtime plugins lightweight and efficient
+2. Use descriptive naming conventions
+3. Implement graceful error handling
+4. Include shebang for portability
+5. Validate configuration inputs
+6. Add timeout mechanisms for long operations

docs/source/developer_guide/extending_lmcache/storage_plugins.rst

@@ -0,0 +1,248 @@
+Storage Plugins
+===============
+
+LMCache supports out of the box storage backends like Mooncake, S3 and NIXL.
+The LMCache storage plugin system provides the ability to add custom storage backends through dynamic loading or plug and play capability. In other words, extending cache storage capabilities without modifying core code.
+
+Backend Definition Requirements
+-------------------------------
+1. Inherit from ``StoragePluginInterface``
+2. Implement all the abstract methods of the parent interface of ``StoragePluginInterface``- ``StorageBackendInterface``
+3. Package as an installable Python module
+
+.. note::
+
+  The interface constructor is the instantiation contract that the LMCache loading system will use when loading custom storage backends.
+  If you wish to implement a constructor, it should have the same parameter signature and call the interface constructor.
+
+How to Integrate the Backend with LMCache
+-----------------------------------------
+1. Install your backend package in the LMCache environment
+2. Add ``storage_plugins`` and its related ``module_path`` and ``class_name`` to ``extra_config`` section of LMCache configuration as follows:
+
+.. code-block:: yaml
+
+    chunk_size: 64
+    local_cpu: False
+    max_local_cpu_size: 5
+    storage_plugins: <backend_name>
+    extra_config:
+      storage_plugin.<backend_name>.module_path: <module_path>
+      storage_plugin.<backend_name>.class_name: <class_name>
+
+An example configuration for a logging backend is as follows:
+
+.. code-block:: yaml
+
+    chunk_size: 64
+    local_cpu: False
+    max_local_cpu_size: 5
+    storage_plugins: "log_backend"
+    extra_config:
+      storage_plugin.log_backend.module_path: lmc_external_log_backend.lmc_external_log_backend
+      storage_plugin.log_backend.class_name: ExternalLogBackend
+
+.. note::
+
+   - Storage backends are initialized in order during LMCache startup - earlier backends have higher priority during cache lookups
+   - ``storage_plugin.<backend_name>`` distinguishes the different dynamic loaded backends
+
+Backend Implementation Example
+------------------------------
+A sample custom backend implementation can be viewed at https://github.com/opendataio/lmc_external_log_backend/
+
+
+MP-Mode L2 Adapter Plugins (``plugin``)
+----------------------------------------
+
+.. _plugin-l2-adapter-overview:
+
+The storage plugin system described above applies to **non-MP mode** (single-process). For
+**MP mode** (multiprocess), LMCache provides the ``plugin`` L2 adapter type, which dynamically
+loads a third-party ``L2AdapterInterface`` implementation at runtime.
+
+Overview
+~~~~~~~~
+
+The ``plugin`` adapter type lets you ship a full L2 adapter as a **separate, pip-installable
+package**. At startup, LMCache imports your module, instantiates your adapter class, and uses it
+just like a built-in adapter -- no LMCache source modifications required.
+
+.. note::
+
+   If your storage backend is a native C++ connector (pybind-wrapped), consider using the
+   ``native_plugin`` type instead (see :doc:`native_connectors`). It reuses the built-in
+   bridging logic so you only need to implement 6 connector methods rather than the full
+   ``L2AdapterInterface``.
+
+Configuration
+~~~~~~~~~~~~~
+
+.. code-block:: json
+
+    {
+      "type": "plugin",
+      "module_path": "my_plugin.adapter",
+      "class_name": "MyL2Adapter",
+      "adapter_params": {
+        "host": "localhost",
+        "capacity": 1000
+      }
+    }
+
+.. list-table:: ``PluginL2AdapterConfig`` fields
+   :header-rows: 1
+   :widths: 22 10 10 58
+
+   * - Field
+     - Type
+     - Required
+     - Description
+   * - ``module_path``
+     - ``str``
+     - yes
+     - Dotted Python import path of the module containing the adapter class.
+   * - ``class_name``
+     - ``str``
+     - yes
+     - Name of the class inside ``module_path`` that implements ``L2AdapterInterface``.
+   * - ``adapter_params``
+     - ``dict``
+     - no
+     - Forwarded to the adapter class constructor (as a typed config or raw dict).
+   * - ``config_class_name``
+     - ``str``
+     - no
+     - Explicit config class name; when omitted the factory auto-discovers it.
+
+Config Class Auto-Discovery
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The factory automatically resolves the adapter's config class using the following priority
+chain (first match wins):
+
+1. **Explicit** ``config_class_name`` field in JSON config.
+2. **Convention**: adapter class name + ``"Config"`` suffix (e.g. ``MyL2Adapter`` looks for
+   ``MyL2AdapterConfig``).
+3. **Attribute**: ``config_class_name`` attribute on the adapter class itself.
+4. **Fallback**: no config class found -- pass raw ``adapter_params`` dict.
+
+Each candidate is looked up in the loaded module and validated as an ``L2AdapterConfigBase``
+subclass. This means most plugins that follow the naming convention will **automatically**
+receive a typed config instance without any extra configuration.
+
+Plugin Contract
+~~~~~~~~~~~~~~~
+
+A plugin adapter class **must**:
+
+1. Subclass ``L2AdapterInterface`` from ``lmcache.v1.distributed.l2_adapters.base``.
+2. Implement all abstract methods: ``submit_store_task``, ``pop_completed_store_tasks``,
+   ``submit_lookup_and_lock_task``, ``query_lookup_and_lock_result``, ``submit_unlock``,
+   ``submit_load_task``, ``query_load_result``, ``close``, and all three event-fd getters.
+3. Provide **three distinct event fds** (store / lookup / load). The controllers build
+   ``fd -> adapter`` maps; duplicates will misroute events.
+4. Be **thread-safe**: ``StoreController`` and ``PrefetchController`` call adapter methods
+   from different threads concurrently.
+5. Accept ``**kwargs`` in ``__init__`` to stay forward-compatible.
+
+A plugin adapter class **should**:
+
+1. Create its own asyncio event loop and background thread if it needs async I/O.
+2. Use ``create_event_notifier()`` from ``lmcache.v1.platform`` for the three event fds
+   (cross-platform: ``os.eventfd`` on Linux, ``os.pipe`` fallback elsewhere).
+3. Clean up all resources (event fds, threads, connections) in ``close()``.
+
+Loading Flow
+~~~~~~~~~~~~
+
+.. code-block:: text
+
+    CLI / config JSON
+      |
+      v
+    PluginL2AdapterConfig.from_dict(d)
+      |  validates module_path, class_name, adapter_params
+      |
+      v
+    _create_plugin_adapter(config, ...)
+      |
+      +-- importlib.import_module(config.module_path)
+      +-- getattr(module, config.class_name)
+      +-- issubclass check against L2AdapterInterface
+      |
+      +-- _resolve_config_class(module, config, adapter_cls)
+      |   +-- 1. config.config_class_name (explicit)
+      |   +-- 2. class_name + "Config" (convention)
+      |   +-- 3. adapter_cls.config_class_name (attribute)
+      |   +-- 4. None (fall back to raw dict)
+      |
+      +-- [if config class found]
+      |   +-- adapter_cls(cfg_cls.from_dict(adapter_params))
+      |
+      +-- [otherwise]
+          +-- adapter_cls(adapter_params)
+              |
+              v
+      L2AdapterInterface instance (ready for use)
+
+Minimal Example
+~~~~~~~~~~~~~~~
+
+.. code-block:: python
+
+    # my_plugin/adapter.py
+    import asyncio
+    import threading
+
+    from lmcache.native_storage_ops import Bitmap
+    from lmcache.v1.distributed.l2_adapters.base import (
+        L2AdapterInterface,
+        L2TaskId,
+    )
+    from lmcache.v1.platform import create_event_notifier
+
+
+    class MyL2Adapter(L2AdapterInterface):
+        def __init__(self, params, **_kw):
+            self._store_efd = create_event_notifier()
+            self._lookup_efd = create_event_notifier()
+            self._load_efd = create_event_notifier()
+            # ... set up connection, background thread, etc.
+
+        # implement all abstract methods ...
+
+        def close(self) -> None:
+            self._store_efd.close()
+            self._lookup_efd.close()
+            self._load_efd.close()
+
+Launch via CLI:
+
+.. code-block:: bash
+
+    --l2-adapter '{
+      "type": "plugin",
+      "module_path": "my_plugin.adapter",
+      "class_name": "MyL2Adapter",
+      "adapter_params": {"host": "localhost"}
+    }'
+
+Reference Implementation
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See ``examples/lmc_external_l2_adapter/`` for a complete, pip-installable example plugin
+(``InMemoryL2Adapter``) that demonstrates:
+
+- FIFO eviction with configurable capacity.
+- Simulated bandwidth delay for realistic testing.
+- Background asyncio event loop with proper shutdown.
+- Full test suite covering store, lookup, load, batch operations, and eviction behavior.
+
+Additional Resources
+~~~~~~~~~~~~~~~~~~~~
+
+- Plugin adapter source: ``lmcache/v1/distributed/l2_adapters/plugin_l2_adapter.py``
+- Native plugin adapter: ``lmcache/v1/distributed/l2_adapters/native_connector_l2_adapter.py``
+- Design document: ``lmcache/v1/distributed/l2_adapters/design_docs/plugin.md``
+- L2 adapter base interface: ``lmcache/v1/distributed/l2_adapters/base.py``