chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,97 @@
|
||||
lmcache coordinator
|
||||
===================
|
||||
|
||||
The ``lmcache coordinator`` command launches the LMCache MP **coordinator**, a
|
||||
standalone HTTP service that tracks the MP server instances in a deployment. MP
|
||||
servers register with it and send periodic heartbeats; the coordinator evicts
|
||||
any instance whose heartbeat lapses past ``--instance-timeout``.
|
||||
|
||||
It replaces ``python -m lmcache.v1.mp_coordinator``. The process runs in the
|
||||
foreground; stop it with ``Ctrl-C``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache coordinator [options]
|
||||
|
||||
Quick start
|
||||
-----------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache coordinator \
|
||||
--host 0.0.0.0 --port 9300 \
|
||||
--instance-timeout 30 \
|
||||
--health-check-interval 10
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 35 65
|
||||
|
||||
* - Flag
|
||||
- Description
|
||||
* - ``--host HOST``
|
||||
- Bind address for the coordinator's HTTP server (default: ``0.0.0.0``).
|
||||
* - ``--port PORT``
|
||||
- HTTP port (default: ``9300``).
|
||||
* - ``--instance-timeout SECS``
|
||||
- Seconds without a heartbeat after which an instance is evicted
|
||||
(default: ``30``).
|
||||
* - ``--health-check-interval SECS``
|
||||
- Seconds between health-check sweeps; ``0`` disables the loop
|
||||
(default: ``10``).
|
||||
* - ``--eviction-check-interval SECS``
|
||||
- Seconds between L2 eviction sweeps; ``0`` disables the loop
|
||||
(default: ``5``).
|
||||
* - ``--eviction-ratio RATIO``
|
||||
- Fraction of tracked keys (by count) to evict per cycle, ``0.0`` to
|
||||
``1.0`` (default: ``0.2``).
|
||||
* - ``--trigger-watermark RATIO``
|
||||
- Eviction fires when usage reaches this fraction of the quota, ``0.0``
|
||||
(exclusive) to ``1.0`` (default: ``1.0``).
|
||||
* - ``--chunk-size N``
|
||||
- Tokens per KV chunk: the CacheBlend match unit and the unit used to
|
||||
resolve pin ``token_ids`` to keys. Must equal the MP servers'
|
||||
``--chunk-size`` (default: ``256``).
|
||||
* - ``--hash-algorithm NAME``
|
||||
- Token hash algorithm for pin key resolution; must equal the MP servers'
|
||||
``--hash-algorithm``. ``blake3`` (default) is self-contained; other
|
||||
algorithms require vLLM importable in the coordinator.
|
||||
* - ``--blend-probe-stride N``
|
||||
- Positions between CacheBlend match probes; ``1`` probes every offset
|
||||
for full recall (default: ``1``).
|
||||
* - ``--timeout-keep-alive SECS``
|
||||
- Seconds the HTTP server keeps idle connections open before closing
|
||||
them. Must be greater than the MP servers' heartbeat interval
|
||||
(default ``5``), otherwise heartbeat requests may hit a closing
|
||||
connection and fail with ``Server disconnected without sending a
|
||||
response`` (default: ``10``).
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Every flag is optional. Unset flags fall back to the
|
||||
``LMCACHE_MP_COORDINATOR_*`` environment variables (``HOST``, ``PORT``,
|
||||
``INSTANCE_TIMEOUT``, ``HEALTH_CHECK_INTERVAL``, ``EVICTION_CHECK_INTERVAL``,
|
||||
``EVICTION_RATIO``, ``TRIGGER_WATERMARK``, ``CHUNK_SIZE``, ``HASH_ALGORITHM``,
|
||||
``BLEND_PROBE_STRIDE``, ``TIMEOUT_KEEP_ALIVE``), and then to the built-in
|
||||
defaults. A supplied flag always overrides the matching env-derived value, so
|
||||
env-only deployments keep working unchanged.
|
||||
|
||||
A second set of env-only knobs controls the startup L2 resync —
|
||||
``LMCACHE_MP_COORDINATOR_ENABLE_STARTUP_RESYNC`` (default ``True``),
|
||||
``LMCACHE_MP_COORDINATOR_RESYNC_POLL_INTERVAL`` (``1``),
|
||||
``LMCACHE_MP_COORDINATOR_RESYNC_MAX_WAIT`` (``60``), and
|
||||
``LMCACHE_MP_COORDINATOR_RESYNC_PAGE_SIZE`` (``1000``). See
|
||||
:doc:`/mp/coordinator` for the boot-time resync flow and the active
|
||||
eviction loop.
|
||||
|
||||
The coordinator drives fleet-wide L2 eviction by calling each MP
|
||||
server's ``DELETE /l2`` endpoint, and resync paginates ``GET /l2/keys``
|
||||
on a registered MP server. Both endpoints are documented at
|
||||
:ref:`mp-http-l2-keys-api`.
|
||||
|
||||
See :doc:`/mp/coordinator` for the coordinator's architecture, registration
|
||||
protocol, and HTTP API.
|
||||
@@ -0,0 +1,218 @@
|
||||
lmcache describe
|
||||
================
|
||||
|
||||
The ``lmcache describe`` command shows the detailed status of a running
|
||||
service. Two targets are supported:
|
||||
|
||||
- ``kvcache`` — the LMCache KV cache service (health, L1 storage, registered
|
||||
models, L2 adapters).
|
||||
- ``engine`` — the inference engine (vLLM) LMCache is paired with (model,
|
||||
context window, health, in-flight requests).
|
||||
|
||||
KV Cache Service (``kvcache``)
|
||||
------------------------------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache describe kvcache --url http://localhost:8000
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
============ LMCache KV Cache Service ============
|
||||
Health: OK
|
||||
URL: http://localhost:8000
|
||||
Engine type: BlendEngine
|
||||
Chunk size: 256
|
||||
L1 capacity (GB): 60.00
|
||||
L1 used (GB): 42.30 (70.5%)
|
||||
Eviction policy: LRU
|
||||
Cached objects: 1024
|
||||
Active sessions: 3
|
||||
---- Model: meta-llama/Llama-3.1-70B-Instruct ----
|
||||
Model: meta-llama/Llama-3.1-70B-Instruct
|
||||
World size: 4
|
||||
GPU IDs: 0, 1, 2, 3
|
||||
Num layers: 80
|
||||
Num blocks: 2048
|
||||
Cache size per token (bytes): 327680
|
||||
--- Kernel group 0 (meta-llama/Llama-3.1-70B-Instruct) ---
|
||||
Kernel group index: 0
|
||||
Engine group index: 0
|
||||
Object group index: 0
|
||||
Num layers: 80
|
||||
Slots per block: 128
|
||||
Dtype: torch.float16
|
||||
MLA: False
|
||||
Attention backend: vLLM non-MLA flash attention
|
||||
Engine KV shape: NL x [2, NB, BS, NH, HS]
|
||||
Engine KV tensor shape: 80 x [2, 2048, 128, 8, 128]
|
||||
------------- L2: NixlStoreL2Adapter -------------
|
||||
Type: NixlStoreL2Adapter
|
||||
Health: OK
|
||||
Backend: nixl_rdma
|
||||
Stored objects: 512
|
||||
Pool used: 480 / 512 (93.8%)
|
||||
==================================================
|
||||
|
||||
The output shows:
|
||||
|
||||
- **Overview** — health status, engine type, chunk size.
|
||||
- **L1 storage** — capacity, usage, eviction policy, cached object count.
|
||||
- **Registered models** — per-model KV cache layout: a context-wide summary
|
||||
followed by one kernel group section per kernel group, each with the engine
|
||||
KV tensor shape (symbolic and concrete), attention backend, and group geometry.
|
||||
- **L2 adapters** — type, health, backend, stored objects, and utilization.
|
||||
|
||||
Inference Engine (``engine``)
|
||||
-----------------------------
|
||||
|
||||
``describe engine`` inspects the vLLM inference engine instead of the LMCache
|
||||
service, reading only the engine's own HTTP endpoints (``/v1/models``,
|
||||
``/health``, ``/metrics``).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache describe engine --url http://localhost:8000
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
================ Inference Engine ================
|
||||
Model: meta-llama/Llama-3.1-8B-Instruct
|
||||
Max context (tokens): 131072
|
||||
Status: OK
|
||||
Running requests: 3
|
||||
==================================================
|
||||
|
||||
The output shows:
|
||||
|
||||
- **Model** and **Max context** — the served model id and its maximum context
|
||||
length, from ``/v1/models``.
|
||||
- **Status** — ``OK`` / ``UNHEALTHY`` from the engine's ``/health`` probe.
|
||||
- **Running requests** — in-flight requests, summed from the
|
||||
``vllm:num_requests_running`` metric. Shows ``N/A`` if metrics are disabled
|
||||
or unreachable.
|
||||
|
||||
Only the ``/v1/models`` fetch is required: if ``/health`` or ``/metrics`` is
|
||||
unavailable the command still reports what it can rather than failing.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache describe engine --url http://localhost:8000 --format json
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"title": "Inference Engine",
|
||||
"metrics": {
|
||||
"model": "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"max_context": 131072,
|
||||
"status": "OK",
|
||||
"running_requests": 3
|
||||
}
|
||||
}
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 75
|
||||
|
||||
* - Flag
|
||||
- Description
|
||||
* - ``target``
|
||||
- What to describe (positional, required): ``kvcache`` or ``engine``.
|
||||
* - ``--url``
|
||||
- Server URL. Defaults per target: ``http://localhost:8080`` for
|
||||
``kvcache``, ``http://localhost:8000`` for ``engine``.
|
||||
* - ``--format``
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output PATH``
|
||||
- Save metrics to a file (format follows ``--format``).
|
||||
* - ``-q`` / ``--quiet``
|
||||
- Suppress stdout output. Exit code only.
|
||||
|
||||
JSON Output
|
||||
-----------
|
||||
|
||||
Use ``--format json`` for machine-readable output. Models, kernel groups, and
|
||||
L2 adapters are collected into lists for easy programmatic access:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache describe kvcache --url http://localhost:8000 --format json
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"title": "LMCache KV Cache Service",
|
||||
"metrics": {
|
||||
"health": "OK",
|
||||
"url": "http://localhost:8000",
|
||||
"engine_type": "BlendEngine",
|
||||
"chunk_size": 256,
|
||||
"l1_capacity_gb": 60.0,
|
||||
"l1_used_gb": "42.30 (70.5%)",
|
||||
"eviction_policy": "LRU",
|
||||
"cached_objects": 1024,
|
||||
"active_sessions": 3,
|
||||
"models": [
|
||||
{
|
||||
"model": "meta-llama/Llama-3.1-70B-Instruct",
|
||||
"world_size": 4,
|
||||
"gpu_ids": "0, 1, 2, 3",
|
||||
"num_layers": 80,
|
||||
"num_blocks": 2048,
|
||||
"cache_size_per_token": 327680
|
||||
}
|
||||
],
|
||||
"kernel_groups": [
|
||||
{
|
||||
"model": "meta-llama/Llama-3.1-70B-Instruct",
|
||||
"kernel_group_idx": 0,
|
||||
"engine_group_idx": 0,
|
||||
"object_group_idx": 0,
|
||||
"num_layers": 80,
|
||||
"slots_per_block": 128,
|
||||
"dtype": "torch.float16",
|
||||
"is_mla": false,
|
||||
"attention_backend": "vLLM non-MLA flash attention",
|
||||
"engine_kv_shape": "NL x [2, NB, BS, NH, HS]",
|
||||
"engine_kv_concrete_shape": "80 x [2, 2048, 128, 8, 128]"
|
||||
}
|
||||
],
|
||||
"l2_adapters": [
|
||||
{
|
||||
"type": "NixlStoreL2Adapter",
|
||||
"health": "OK",
|
||||
"backend": "nixl_rdma",
|
||||
"stored_object_count": 512,
|
||||
"pool_used": "480 / 512 (93.8%)"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
Engine KV Shape Abbreviations
|
||||
-----------------------------
|
||||
|
||||
The ``engine_kv_shape`` field uses short names from the ``EngineKVFormat`` enum:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 15 85
|
||||
|
||||
* - Abbrev
|
||||
- Meaning
|
||||
* - NB
|
||||
- num_blocks
|
||||
* - NL
|
||||
- num_layers
|
||||
* - BS
|
||||
- block_size
|
||||
* - NH
|
||||
- num_heads
|
||||
* - HS
|
||||
- head_size
|
||||
* - PBS
|
||||
- page_buffer_size (NB × BS)
|
||||
@@ -0,0 +1,108 @@
|
||||
CLI Reference
|
||||
=============
|
||||
|
||||
The ``lmcache`` command-line interface provides tools for launching,
|
||||
managing, inspecting, and benchmarking LMCache servers and the inference
|
||||
engines in front of them.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache <command> [options]
|
||||
|
||||
After installing LMCache, the ``lmcache`` command is available globally.
|
||||
Run ``lmcache -h`` to see all commands, or ``lmcache <command> -h`` for a
|
||||
specific command.
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
The ``lmcache`` CLI ships in two packages:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 30 45
|
||||
|
||||
* - Package
|
||||
- Install
|
||||
- When to use
|
||||
* - ``lmcache``
|
||||
- ``pip install lmcache``
|
||||
- Full install: server, CLI, and CUDA extensions. Required for
|
||||
``server``, ``bench server``, ``bench l2``, and ``trace``.
|
||||
Linux + GPU.
|
||||
* - ``lmcache-cli``
|
||||
- ``pip install lmcache-cli``
|
||||
- CLI only: ``ping``, ``query``, ``describe``, ``kvcache``,
|
||||
``quota``, ``bench engine``. No GPU required, any OS.
|
||||
|
||||
.. note::
|
||||
|
||||
Do not install both packages in the same environment — they both provide
|
||||
the ``lmcache`` entry point.
|
||||
|
||||
Available Commands
|
||||
------------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 80
|
||||
|
||||
* - Command
|
||||
- Description
|
||||
* - :doc:`server`
|
||||
- Launch the LMCache MP server (ZMQ + HTTP). Requires the full install.
|
||||
* - :doc:`coordinator`
|
||||
- Launch the LMCache MP coordinator (HTTP instance registry).
|
||||
* - :doc:`describe`
|
||||
- Show detailed status of a running LMCache service.
|
||||
* - :doc:`ping`
|
||||
- Liveness check for LMCache or vLLM servers.
|
||||
* - :doc:`query`
|
||||
- Single-shot query interface for the serving engine.
|
||||
* - :doc:`bench`
|
||||
- Run sustained benchmarks against an inference engine
|
||||
(``engine``), an LMCache MP server (``server``), or an L2 cache
|
||||
adapter (``l2``).
|
||||
* - :doc:`kvcache`
|
||||
- Manage KV cache state (e.g. clear L1 cache) on a running server.
|
||||
* - :doc:`quota`
|
||||
- Manage per-salt cache quotas (set, get, list, delete).
|
||||
* - :doc:`trace`
|
||||
- Inspect and replay storage-level trace files.
|
||||
* - :doc:`tool`
|
||||
- Run offline analysis tools (e.g. the cache simulator).
|
||||
|
||||
Output Formats
|
||||
--------------
|
||||
|
||||
Commands that produce metrics share three common flags:
|
||||
|
||||
* ``--format {terminal,json}`` — stdout format (default: ``terminal``).
|
||||
* ``--output PATH`` — also write metrics to a file (uses ``--format``).
|
||||
* ``-q`` / ``--quiet`` — suppress stdout; rely on the exit code.
|
||||
|
||||
The terminal output uses human-readable labels (e.g. ``"Round trip time
|
||||
(ms)"``), while JSON uses machine-readable keys (e.g.
|
||||
``"round_trip_time_ms"``).
|
||||
|
||||
Adding New Commands
|
||||
-------------------
|
||||
|
||||
New CLI subcommands are added by creating a ``BaseCommand`` subclass under
|
||||
``lmcache/cli/commands/``; they are discovered and registered automatically.
|
||||
See :doc:`/developer_guide/cli` for details.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
server
|
||||
coordinator
|
||||
describe
|
||||
ping
|
||||
query
|
||||
bench
|
||||
kvcache
|
||||
quota
|
||||
trace
|
||||
tool
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
lmcache kvcache
|
||||
===============
|
||||
|
||||
The ``lmcache kvcache`` command manages KV cache state on a running LMCache
|
||||
server.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache kvcache <sub-command> [options]
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
$ lmcache kvcache -h
|
||||
usage: lmcache kvcache [-h] [--format FORMAT] [--output PATH] [-q] {clear} ...
|
||||
|
||||
Manage KV cache state.
|
||||
|
||||
subcommands:
|
||||
clear Clear all cached KV data in L1 (CPU)
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
--format FORMAT Stdout output format (default: terminal). Available: terminal, json.
|
||||
--output PATH Save metrics to a file at PATH (format chosen by --format).
|
||||
-q, --quiet Suppress stdout output. Exit code only.
|
||||
|
||||
clear
|
||||
-----
|
||||
|
||||
Clear all cached KV data in **L1 (CPU memory)** on the target LMCache server.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache kvcache clear --url <MP_HTTP_URL>
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache kvcache clear --url http://localhost:8000
|
||||
|
||||
================ KV Cache Clear ================
|
||||
Status: OK
|
||||
================================================
|
||||
|
||||
**JSON output** (for scripting with ``jq``):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache kvcache clear --url http://localhost:8000 --format json
|
||||
{
|
||||
"title": "KV Cache Clear",
|
||||
"metrics": {
|
||||
"status": "OK"
|
||||
}
|
||||
}
|
||||
|
||||
**Quiet mode** (exit code only, no output):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache kvcache clear -q --url http://localhost:8000
|
||||
$ echo $?
|
||||
0
|
||||
|
||||
Options
|
||||
~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 10 65
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``--url``
|
||||
- Yes
|
||||
- URL of the LMCache MP HTTP server (e.g. ``http://localhost:8000``).
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output``
|
||||
- No
|
||||
- Save output to a file (uses the format chosen by ``--format``).
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout. Useful in scripts where you only need the exit code.
|
||||
|
||||
Exit Codes
|
||||
~~~~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 15 85
|
||||
|
||||
* - Code
|
||||
- Meaning
|
||||
* - ``0``
|
||||
- Success.
|
||||
* - ``1``
|
||||
- Error (connection failure, server error, bad arguments).
|
||||
|
||||
Common Patterns
|
||||
---------------
|
||||
|
||||
**Handle temporary server unavailability:**
|
||||
|
||||
If the server is temporarily unreachable (e.g. due to network issue), the command
|
||||
fails with exit code 1. For persistent connectivity issues, use ``lmcache ping``
|
||||
to diagnose.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
if lmcache kvcache clear -q --url http://localhost:8000; then
|
||||
echo "Cache cleared"
|
||||
else
|
||||
echo "Clear failed — server temporarily unreachable, retrying later"
|
||||
fi
|
||||
|
||||
**Clear cache and capture JSON result:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
RESULT=$(lmcache kvcache clear --url http://localhost:8000 --format json)
|
||||
STATUS=$(echo "$RESULT" | jq -r '.metrics.status')
|
||||
echo "Clear status: $STATUS"
|
||||
@@ -0,0 +1,75 @@
|
||||
lmcache ping
|
||||
============
|
||||
|
||||
The ``lmcache ping`` command is a liveness check for an LMCache KV cache
|
||||
server or a vLLM serving engine.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Ping the KV cache server (default: http://localhost:8080)
|
||||
lmcache ping kvcache
|
||||
|
||||
# Ping the serving engine (default: http://localhost:8000)
|
||||
lmcache ping engine --url http://localhost:8000
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
======= Ping KV Cache ========
|
||||
Status: OK
|
||||
Round trip time (ms): 3.42
|
||||
==============================
|
||||
|
||||
``ping kvcache`` checks the ``/healthcheck`` endpoint; ``ping engine`` checks
|
||||
``/health``.
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 75
|
||||
|
||||
* - Flag
|
||||
- Description
|
||||
* - ``kvcache`` | ``engine``
|
||||
- Target to ping (positional, required).
|
||||
* - ``--url``
|
||||
- Server URL. Defaults to ``http://localhost:8080`` for ``kvcache``,
|
||||
``http://localhost:8000`` for ``engine``.
|
||||
* - ``--format``
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output PATH``
|
||||
- Save metrics to a file (format follows ``--format``).
|
||||
* - ``-q`` / ``--quiet``
|
||||
- Suppress stdout output. Exit code only.
|
||||
|
||||
JSON Output
|
||||
-----------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache ping kvcache --format json
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"title": "Ping KV Cache",
|
||||
"metrics": {
|
||||
"status": "OK",
|
||||
"round_trip_time_ms": 3.42
|
||||
}
|
||||
}
|
||||
|
||||
Exit Codes
|
||||
----------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 15 85
|
||||
|
||||
* - Code
|
||||
- Meaning
|
||||
* - ``0``
|
||||
- Server is reachable (HTTP 200).
|
||||
* - ``1``
|
||||
- Connection failure or non-200 response.
|
||||
@@ -0,0 +1,90 @@
|
||||
lmcache query
|
||||
=============
|
||||
|
||||
The ``lmcache query`` command sends a single OpenAI-compatible inference
|
||||
request and reports token and latency metrics. It has two targets:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache query {engine,kvcache} [options]
|
||||
|
||||
* ``engine`` — send one request to a serving engine's HTTP API.
|
||||
* ``kvcache`` — query KV-cache endpoints (not implemented yet).
|
||||
|
||||
|
||||
query engine
|
||||
------------
|
||||
|
||||
The ``query engine`` subcommand sends one request to the engine API and
|
||||
reports metrics. ``--prompt`` supports placeholders: ``{lmcache}`` loads
|
||||
``lmcache/cli/documents/lmcache.txt``, and custom documents can be passed with
|
||||
``--documents NAME=PATH``. The prompt token count is taken directly from the
|
||||
usage data reported by the engine (``stream_options: {include_usage: true}``).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache query engine --url http://localhost:8000/v1 \
|
||||
--prompt "{lmcache} Summarize LMCache usage." \
|
||||
--format terminal \
|
||||
--max-tokens 128
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
================= Query Engine =================
|
||||
Model: facebook/opt-125m
|
||||
Input tokens: 618
|
||||
--------------- Latency Metrics ----------------
|
||||
Output tokens: 9
|
||||
TTFT (ms): 26.88
|
||||
TPOT (ms/token): 0.91
|
||||
Total latency (ms): 35.05
|
||||
Throughput (tokens/s): 1100.64
|
||||
================================================
|
||||
|
||||
Options
|
||||
~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 15 55
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``--url URL``
|
||||
- Yes
|
||||
- Serving engine base URL (e.g. ``http://localhost:8000/v1``).
|
||||
* - ``--prompt TEXT``
|
||||
- Yes
|
||||
- Prompt text with optional ``{name}`` placeholders. ``{lmcache}``
|
||||
expands to the bundled sample document.
|
||||
* - ``--model ID``
|
||||
- No
|
||||
- Model ID for the serving engine. Auto-detected from the engine's
|
||||
reported usage if omitted.
|
||||
* - ``--max-tokens N``
|
||||
- No
|
||||
- Maximum completion tokens (default: 128).
|
||||
* - ``--timeout SECS``
|
||||
- No
|
||||
- HTTP timeout in seconds (default: 30).
|
||||
* - ``--documents NAME=PATH``
|
||||
- No
|
||||
- Load file text for ``{NAME}`` in ``--prompt``. Accepts one or more
|
||||
``NAME=PATH`` values.
|
||||
* - ``--completions``
|
||||
- No
|
||||
- Use ``POST /v1/completions`` only.
|
||||
* - ``--chat-first``
|
||||
- No
|
||||
- Try ``/v1/chat/completions`` first, then fall back to
|
||||
``/v1/completions``.
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output PATH``
|
||||
- No
|
||||
- Save metrics to a file (format follows ``--format``).
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout output. Exit code only.
|
||||
@@ -0,0 +1,301 @@
|
||||
lmcache quota
|
||||
=============
|
||||
|
||||
The ``lmcache quota`` command manages per-salt cache quotas on a running
|
||||
LMCache server. Quotas are soft limits: exceeding a quota triggers eviction
|
||||
on the next cycle (~1 s) rather than rejecting writes.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache quota <sub-command> [options]
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
$ lmcache quota -h
|
||||
usage: lmcache quota [-h] {set,get,list,delete} ...
|
||||
|
||||
Manage per-salt cache quotas on the LMCache server.
|
||||
|
||||
subcommands:
|
||||
set Create or update a quota for a cache_salt
|
||||
get Show the quota and current usage for a cache_salt
|
||||
list List all registered quotas and their usage
|
||||
delete Remove a quota for a cache_salt
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
|
||||
set
|
||||
---
|
||||
|
||||
Create or update a quota for a given ``cache_salt``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache quota set <salt> --limit-gb <GB> [--url <URL>]
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache quota set tenant1 --limit-gb 10.5
|
||||
|
||||
================ Quota Set =================
|
||||
Cache salt: tenant1
|
||||
Limit (GB): 10.5
|
||||
Status: ok
|
||||
=============================================
|
||||
|
||||
**Options:**
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 10 65
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``<salt>``
|
||||
- Yes
|
||||
- The ``cache_salt`` identifier. Use ``_default`` for anonymous
|
||||
(un-salted) traffic.
|
||||
* - ``--limit-gb``
|
||||
- Yes
|
||||
- Quota limit in gigabytes (non-negative float).
|
||||
* - ``--url``
|
||||
- No
|
||||
- LMCache HTTP server URL (default: ``http://localhost:8080``).
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output``
|
||||
- No
|
||||
- Save output to a file (uses the format chosen by ``--format``).
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout output. Exit code only.
|
||||
|
||||
get
|
||||
---
|
||||
|
||||
Show the current quota limit and live usage for a specific ``cache_salt``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache quota get <salt> [--url <URL>]
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache quota get tenant1
|
||||
|
||||
================ Quota Info ================
|
||||
Cache salt: tenant1
|
||||
Limit (GB): 10.5
|
||||
Current usage (GB): 3.27
|
||||
Exists: True
|
||||
=============================================
|
||||
|
||||
**Options:**
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 10 65
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``<salt>``
|
||||
- Yes
|
||||
- The ``cache_salt`` identifier.
|
||||
* - ``--url``
|
||||
- No
|
||||
- LMCache HTTP server URL (default: ``http://localhost:8080``).
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output``
|
||||
- No
|
||||
- Save output to a file.
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout output.
|
||||
|
||||
list
|
||||
----
|
||||
|
||||
List all registered quotas along with their current usage.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache quota list [--url <URL>]
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache quota list
|
||||
|
||||
=================== Quota List ====================
|
||||
--- Salt: tenant1 ---
|
||||
Cache salt: tenant1
|
||||
Limit (GB): 10.5
|
||||
Current usage (GB): 3.27
|
||||
--- Salt: _default ---
|
||||
Cache salt: _default
|
||||
Limit (GB): 5.0
|
||||
Current usage (GB): 1.82
|
||||
===================================================
|
||||
|
||||
**JSON output:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache quota list --format json
|
||||
{
|
||||
"title": "Quota List",
|
||||
"sections": {
|
||||
"quota_0": {
|
||||
"label": "Salt: tenant1",
|
||||
"metrics": {
|
||||
"cache_salt": "tenant1",
|
||||
"limit_gb": 10.5,
|
||||
"current_usage_gb": 3.27
|
||||
}
|
||||
},
|
||||
"quota_1": {
|
||||
"label": "Salt: _default",
|
||||
"metrics": {
|
||||
"cache_salt": "_default",
|
||||
"limit_gb": 5.0,
|
||||
"current_usage_gb": 1.82
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
**Options:**
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 10 65
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``--url``
|
||||
- No
|
||||
- LMCache HTTP server URL (default: ``http://localhost:8080``).
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output``
|
||||
- No
|
||||
- Save output to a file.
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout output.
|
||||
|
||||
delete
|
||||
------
|
||||
|
||||
Remove a quota entry for a given ``cache_salt``. Any bytes still cached
|
||||
under this salt become over-budget on the next eviction cycle and will be
|
||||
evicted (effective limit drops to 0).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache quota delete <salt> [--url <URL>]
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ lmcache quota delete tenant1
|
||||
|
||||
============== Quota Delete ===============
|
||||
Cache salt: tenant1
|
||||
Status: removed
|
||||
===========================================
|
||||
|
||||
**Options:**
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 10 65
|
||||
|
||||
* - Flag
|
||||
- Required
|
||||
- Description
|
||||
* - ``<salt>``
|
||||
- Yes
|
||||
- The ``cache_salt`` identifier.
|
||||
* - ``--url``
|
||||
- No
|
||||
- LMCache HTTP server URL (default: ``http://localhost:8080``).
|
||||
* - ``--format``
|
||||
- No
|
||||
- Output format: ``terminal`` (default) or ``json``.
|
||||
* - ``--output``
|
||||
- No
|
||||
- Save output to a file.
|
||||
* - ``-q`` / ``--quiet``
|
||||
- No
|
||||
- Suppress stdout output.
|
||||
|
||||
Exit Codes
|
||||
----------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 15 85
|
||||
|
||||
* - Code
|
||||
- Meaning
|
||||
* - ``0``
|
||||
- Success.
|
||||
* - ``1``
|
||||
- Error (connection failure, server error, bad arguments).
|
||||
|
||||
The ``_default`` Salt
|
||||
---------------------
|
||||
|
||||
The LMCache server uses an empty string (``""``) as the ``cache_salt`` for
|
||||
anonymous / un-salted traffic. Since empty strings cannot appear in URL path
|
||||
parameters, the HTTP API (and this CLI) uses the sentinel ``_default`` in
|
||||
its place.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Set a 5 GB quota for anonymous traffic
|
||||
lmcache quota set _default --limit-gb 5.0
|
||||
|
||||
# Check usage
|
||||
lmcache quota get _default
|
||||
|
||||
Common Patterns
|
||||
---------------
|
||||
|
||||
**Provision quotas for multiple tenants:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
for tenant in tenant_a tenant_b tenant_c; do
|
||||
lmcache quota set "$tenant" --limit-gb 8.0
|
||||
done
|
||||
|
||||
**Monitor usage in a script:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
USAGE=$(lmcache quota get tenant1 --format json | jq '.metrics.current_usage_gb')
|
||||
LIMIT=$(lmcache quota get tenant1 --format json | jq '.metrics.limit_gb')
|
||||
echo "tenant1: ${USAGE} / ${LIMIT} GB"
|
||||
|
||||
**Revoke access (evict all cached data for a salt):**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Deleting the quota causes all data under this salt to be evicted
|
||||
lmcache quota delete tenant1
|
||||
@@ -0,0 +1,103 @@
|
||||
lmcache server
|
||||
==============
|
||||
|
||||
The ``lmcache server`` command launches the standalone LMCache
|
||||
Multi-Process (MP) server, which exposes a ZMQ control plane and an HTTP
|
||||
frontend (status, healthcheck, cache-clear, checksum APIs). It is the server
|
||||
that ``lmcache describe``, ``lmcache ping kvcache``, ``lmcache kvcache``, and
|
||||
``lmcache bench server`` talk to.
|
||||
|
||||
.. note::
|
||||
|
||||
This command requires the full ``lmcache`` installation with CUDA
|
||||
extensions. It is **not** available in the lightweight ``lmcache-cli``
|
||||
package.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache server [options]
|
||||
|
||||
Quick start
|
||||
-----------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache server \
|
||||
--host 0.0.0.0 --port 5555 \
|
||||
--l1-size-gb 100 \
|
||||
--eviction-policy LRU
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
The server composes its arguments from several configuration modules — the
|
||||
multiprocess server, the storage manager (L1 / L2 adapters / eviction), the
|
||||
HTTP frontend, and the Prometheus / telemetry observability layer. The full,
|
||||
authoritative list is large and evolves with the runtime, so consult:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache server --help
|
||||
|
||||
Commonly used flags include:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 35 65
|
||||
|
||||
* - Flag
|
||||
- Description
|
||||
* - ``--host HOST``
|
||||
- Bind address for the server.
|
||||
* - ``--port PORT``
|
||||
- ZMQ control-plane port.
|
||||
* - ``--chunk-size N``
|
||||
- KV cache chunk size in tokens.
|
||||
* - ``--l1-size-gb GB``
|
||||
- L1 (CPU/DRAM) cache capacity in GB.
|
||||
* - ``--eviction-policy POLICY``
|
||||
- L1 eviction policy (e.g. ``LRU``).
|
||||
* - ``--eviction-trigger-watermark RATIO``
|
||||
- L1 fill ratio at which eviction begins.
|
||||
* - ``--eviction-ratio RATIO``
|
||||
- Fraction of L1 cleared per eviction cycle.
|
||||
* - ``--max-workers N``
|
||||
- Number of server worker processes.
|
||||
* - ``--coordinator-url URL``
|
||||
- Register with an MP coordinator at this base URL (e.g.
|
||||
``http://coordinator:9300``). Opt-in; enables fleet registration. See
|
||||
:doc:`/mp/coordinator`.
|
||||
* - ``--coordinator-advertise-ip IP``
|
||||
- IP the coordinator should reach this server at (defaults to the
|
||||
outbound IP).
|
||||
* - ``--coordinator-heartbeat-interval SECONDS``
|
||||
- Seconds between heartbeats (``> 0``, default ``5``). Keep well below the
|
||||
coordinator's instance timeout.
|
||||
* - ``--coordinator-l2-event-reporting``
|
||||
- Enable reporting L2 store/lookup events to the coordinator for
|
||||
fleet-wide usage tracking and quota-based eviction.
|
||||
* - ``--coordinator-l2-event-flush-interval SECONDS``
|
||||
- Seconds between L2 event batch flushes (``> 0``, default ``1``).
|
||||
* - ``--p2p-advertise-url HOST:PORT``
|
||||
- Enable P2P KV cache sharing and advertise this server's
|
||||
transfer-channel endpoint to peers (e.g. ``10.0.0.1:8500``). Setting it
|
||||
turns P2P on; it additionally requires ``--coordinator-url`` for peer
|
||||
discovery. See :doc:`/mp/p2p`.
|
||||
* - ``--p2p-listen-url HOST:PORT``
|
||||
- Address the transfer-channel server binds to. Defaults to
|
||||
``--p2p-advertise-url``. Set it when the advertised address differs from
|
||||
the bind address (e.g. bind ``0.0.0.0`` while advertising a routable IP).
|
||||
* - ``--p2p-lookup-timeout SECONDS``
|
||||
- Deadline for a peer lookup before it counts as a miss (default ``30``).
|
||||
* - ``--p2p-load-timeout SECONDS``
|
||||
- Deadline for a peer KV read before it counts as a failure
|
||||
(default ``30``).
|
||||
* - ``--p2p-transfer-engine ENGINE``
|
||||
- Transfer-channel implementation for P2P reads (default ``nixl``).
|
||||
* - ``--trace-level {storage}``
|
||||
- Enable storage-level trace recording (see :doc:`trace`).
|
||||
* - ``--trace-output PATH``
|
||||
- Destination for recorded ``.lct`` trace files.
|
||||
|
||||
L2 adapters, observability, and Prometheus exporters are configured through
|
||||
their own flag groups; see ``lmcache server --help`` for the complete set.
|
||||
@@ -0,0 +1,53 @@
|
||||
lmcache tool
|
||||
============
|
||||
|
||||
The ``lmcache tool`` command groups offline analysis utilities bundled with
|
||||
LMCache.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache tool <tool-name> <action> [options]
|
||||
|
||||
Currently one tool is available: ``cache-simulator``.
|
||||
|
||||
.. note::
|
||||
|
||||
``cache-simulator`` depends on the optional ``plot`` extras
|
||||
(``sortedcontainers`` / ``matplotlib``). If they are not installed, the
|
||||
sub-command is silently omitted from the CLI. Install the extras to enable
|
||||
it.
|
||||
|
||||
|
||||
cache-simulator
|
||||
---------------
|
||||
|
||||
Replay LMCache lookup-hash JSONL logs through an LRU cache to measure the
|
||||
KV-cache token hit rate. It has three actions:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache tool cache-simulator {simulate,sweep,gen-dataset} [options]
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 80
|
||||
|
||||
* - Action
|
||||
- Description
|
||||
* - ``simulate``
|
||||
- Replay logs at a fixed cache capacity; print a text report and save a
|
||||
7-panel statistics PNG.
|
||||
* - ``sweep``
|
||||
- Sweep across a range of cache capacities and save a hit-rate vs.
|
||||
capacity PNG.
|
||||
* - ``gen-dataset``
|
||||
- Generate a ``vllm bench serve`` custom dataset (JSONL) from
|
||||
lookup-hash JSONL logs, preserving prefix-sharing structure.
|
||||
|
||||
Each action has its own flags. Run the built-in help for the full list:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache tool cache-simulator simulate --help
|
||||
lmcache tool cache-simulator sweep --help
|
||||
lmcache tool cache-simulator gen-dataset --help
|
||||
@@ -0,0 +1,102 @@
|
||||
lmcache trace
|
||||
=============
|
||||
|
||||
The ``lmcache trace`` command inspects and replays LMCache storage-level
|
||||
trace files (``.lct``). It has two sub-commands:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache trace {info,replay} FILE [options]
|
||||
|
||||
.. note::
|
||||
|
||||
``lmcache trace`` needs the full ``lmcache`` package (StorageManager,
|
||||
trace codecs, ``TraceReader``). It is not available in the lightweight
|
||||
``lmcache-cli`` install and exits with status ``2`` if those modules are
|
||||
missing.
|
||||
|
||||
Trace *capture* is not a ``trace`` sub-command — recording is bound to a live
|
||||
server via ``lmcache server --trace-level storage [--trace-output ...]`` (see
|
||||
:doc:`server`).
|
||||
|
||||
|
||||
info
|
||||
----
|
||||
|
||||
Print a one-screen summary of a trace file: header metadata plus per-qualname
|
||||
record counts.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache trace info path/to/trace.lct
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Trace file: path/to/trace.lct
|
||||
level: storage
|
||||
format_version: 1
|
||||
trace_schema_version: 1
|
||||
duration: 12.345s
|
||||
sm_config_digest: a1b2c3d4
|
||||
total_records: 2048
|
||||
ops:
|
||||
StorageManager.store: 1024
|
||||
StorageManager.retrieve: 1024
|
||||
|
||||
The only argument is the positional ``FILE`` (path to a ``.lct`` trace file).
|
||||
|
||||
|
||||
replay
|
||||
------
|
||||
|
||||
Reissue every recorded call against a fresh ``StorageManager``, honoring the
|
||||
recorded inter-call timings.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache trace replay path/to/trace.lct \
|
||||
--l1-size-gb 10 --eviction-policy LRU
|
||||
|
||||
``replay`` accepts the standard storage-manager configuration flags
|
||||
(``--l1-size-gb``, ``--eviction-policy``, ``--l2-...``); see
|
||||
``lmcache server --help`` for the full list. The replay-side config may
|
||||
differ from the config recorded in the trace, which can legitimately cause
|
||||
retrieve misses.
|
||||
|
||||
.. warning::
|
||||
|
||||
A replay environment mismatch may cause retrieve misses. Replay uses the
|
||||
replay-side StorageManager config (which may differ from the recorded
|
||||
config), runs on a host whose performance may differ from the recording
|
||||
host, and StorageManager reads/writes are async. Treat retrieve-miss
|
||||
counts as a signal about the replay environment, not a defect in the trace.
|
||||
|
||||
Options
|
||||
~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
|
||||
* - Flag
|
||||
- Description
|
||||
* - ``FILE``
|
||||
- Path to a ``.lct`` trace file (positional, required).
|
||||
* - ``--verbose``
|
||||
- Print one line per replayed record.
|
||||
* - ``--jsonl-out PATH``
|
||||
- Write one JSON object per replayed record to ``PATH`` (qualname,
|
||||
latency_ms, failed).
|
||||
* - ``--output-dir DIR``
|
||||
- Directory for aggregated CSV / JSON summary output (default: current
|
||||
directory).
|
||||
* - ``--no-csv``
|
||||
- Skip the aggregated CSV summary export.
|
||||
* - ``--json``
|
||||
- Also export an aggregated JSON summary.
|
||||
* - ``-q`` / ``--quiet``
|
||||
- Suppress the terminal metrics table (files are still written).
|
||||
|
||||
The terminal summary reports overall replay stats (records replayed /
|
||||
skipped / failed, duration, config-digest match) and per-op latency
|
||||
percentiles. ``replay`` exits with status ``1`` if any record failed.
|
||||
Reference in New Issue
Block a user