chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,144 @@
|
||||
Basic Check Tool
|
||||
================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
The LMCache Basic Check Tool is a testing and validation utility that helps you verify your LMCache installation, configuration, and functionality. It provides multiple testing modes to validate different components of the LMCache system.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The basic check tool (``lmcache.v1.basic_check``) is designed to:
|
||||
|
||||
* Test remote backend connectivity and functionality
|
||||
* Validate storage manager operations
|
||||
* Generate test keys for performance testing
|
||||
* Verify configuration settings
|
||||
* Provide diagnostic information for troubleshooting
|
||||
|
||||
Available Check Modes
|
||||
---------------------
|
||||
|
||||
The tool supports several check modes, each targeting specific functionality:
|
||||
|
||||
test_remote
|
||||
~~~~~~~~~~~
|
||||
|
||||
Tests the remote backend functionality including:
|
||||
|
||||
* Connection establishment to remote backends (fs, etc.)
|
||||
* put/get operations with data integrity validation
|
||||
* put/get/exists operations with performance reports
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode test_remote
|
||||
|
||||
test_storage_manager
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tests the storage manager operations including:
|
||||
|
||||
* Configuration validation
|
||||
* batched_put/get operations with data integrity validation
|
||||
* batched_put/get/contains operations with performance reports
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode test_storage_manager
|
||||
|
||||
gen (Key Generation)
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Generates test keys for performance testing and benchmarking:
|
||||
|
||||
* Configurable number of keys and concurrency levels
|
||||
* Memory-efficient batch processing
|
||||
* Progress tracking and performance metrics
|
||||
* Offset support for distributed testing
|
||||
|
||||
**Usage:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode gen --num-keys 1000 --concurrency 16
|
||||
|
||||
Command Line Interface
|
||||
----------------------
|
||||
|
||||
Basic Usage
|
||||
~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode <MODE> [OPTIONS]
|
||||
|
||||
List Available Modes
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.basic_check --mode list
|
||||
|
||||
Command Line Options
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. option:: --mode MODE
|
||||
|
||||
**Required.** Operation mode to run. Use ``list`` to see available modes.
|
||||
|
||||
.. option:: --model MODEL
|
||||
|
||||
Model name for testing, just a part of key of persist kv-cache. Default: ``/lmcache_test_model/``
|
||||
|
||||
.. option:: --num-keys NUM
|
||||
|
||||
Number of keys to generate (gen mode only). Default: 100
|
||||
|
||||
.. option:: --concurrency NUM
|
||||
|
||||
Concurrency level for operations (gen mode only). Default: 16
|
||||
|
||||
.. option:: --offset NUM
|
||||
|
||||
Offset for key generation (gen mode only). Default: 0
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The basic check tool uses your existing LMCache configuration. You can specify configuration in several ways:
|
||||
|
||||
Environment Variable
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export LMCACHE_CONFIG_PATH=/path/to/config.yaml
|
||||
python -m lmcache.v1.basic_check --mode test_remote
|
||||
|
||||
Example Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here's an example configuration optimized for basic checks:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Basic cache settings
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
max_local_cpu_size: 1.0 # 1GB for basic checks
|
||||
|
||||
# Remote backend (optional)
|
||||
remote_url: "file:///tmp/lmcache_basic_check"
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
The ``examples/basic_check/`` directory contains comprehensive examples:
|
||||
@@ -0,0 +1,7 @@
|
||||
Usage Data Module
|
||||
=================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
usage_stats_collection
|
||||
@@ -0,0 +1,155 @@
|
||||
.. _usage-stats-collection:
|
||||
|
||||
Usage Stats Collection
|
||||
======================
|
||||
|
||||
LMCache collects anonymous usage data by default to help the engineering team understand real-world workloads, prioritize optimizations, and improve reliability. All collected data is aggregated and contains no sensitive user information.
|
||||
|
||||
A sanitized subset of the aggregated data may be publicly released for the community’s benefit (for example, see a daily usage report `here <https://github.com/Hanchenli/OSS_Growth_Toolkit/tree/main/usage_tracker/report>`_).
|
||||
|
||||
What data is collected?
|
||||
-----------------------
|
||||
|
||||
Usage stats are implemented in the ``lmcache/usage_telemetry/`` package. The
|
||||
one-shot messages sent at startup depend on how LMCache runs.
|
||||
|
||||
When LMCache runs **inside the serving engine** (the single-process
|
||||
integrations for vLLM, SGLang, and TensorRT-LLM), engine startup sends:
|
||||
|
||||
- **EnvMessage**
|
||||
Captures environment details such as cloud provider, CPU info, total memory, architecture, GPU count/type, and execution source.
|
||||
|
||||
- **EngineMessage**
|
||||
Records engine configuration and metadata, including cache settings (chunk size, local device, cache limits), remote backend parameters, blending settings, model name, world size, and KV-cache dtype/shape.
|
||||
|
||||
- **MetadataMessage**
|
||||
Reports execution metadata: the timestamp when the run started and total duration in seconds.
|
||||
|
||||
When LMCache runs as a **standalone multiprocess (MP) cache server**
|
||||
(``lmcache server``), server startup sends:
|
||||
|
||||
- **EnvMessage**
|
||||
Same environment snapshot as above, taken on the cache-server host.
|
||||
|
||||
- **MPServerMessage**
|
||||
Records the server configuration: LMCache version, chunk size, hash
|
||||
algorithm, engine type, transfer mode, worker pool sizes, whether P2P is
|
||||
enabled, L1 size and medium (DRAM / GDS / DRAM+DevDAX), eviction policy,
|
||||
the configured L2 adapter and serde types, and the L2 store/prefetch
|
||||
policies.
|
||||
|
||||
The MP server's ``--instance-id`` is **never** sent: it can be
|
||||
operator-chosen and therefore identifying. Model names are not known at
|
||||
server startup (vLLM instances register later) and are not part of this
|
||||
message.
|
||||
|
||||
In addition to the one-shot messages above, a continuous reporter periodically
|
||||
sends interval counters (**ContinuousContextMessage**: tokens stored/hit and
|
||||
stored KV bytes in the interval) and a cache-lifespan histogram
|
||||
(**CacheLifespanMessage**). The flush interval is controlled by
|
||||
``LMCACHE_USAGE_TRACK_INTERVAL`` (seconds, default 600).
|
||||
|
||||
Every payload carries four common fields:
|
||||
|
||||
- ``session_id`` -- a random UUID minted once per process, joining the
|
||||
one-shot context with the continuous messages of the same run.
|
||||
- ``machine_id`` -- a random UUID persisted at
|
||||
``~/.config/lmcache/machine_id``, grouping sessions from the same machine.
|
||||
It is never derived from hardware identifiers (MAC address, hostname).
|
||||
- ``schema_version`` -- the version of the message schema.
|
||||
- ``deployment_mode`` -- ``single_process`` (LMCache inside the serving
|
||||
engine) or ``mp_server`` (standalone MP cache server).
|
||||
|
||||
These messages are serialized to JSON and POSTed to the LMCache usage server.
|
||||
|
||||
Example JSON payload
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"message_type": "EnvMessage",
|
||||
"provider": "GCP",
|
||||
"num_cpu": 24,
|
||||
"cpu_type": "Intel(R) Xeon(R) CPU @ 2.20GHz",
|
||||
"cpu_family_model_stepping": "6,85,7",
|
||||
"total_memory": 101261135872,
|
||||
"architecture": ["64bit", "ELF"],
|
||||
"platforms": "Linux-5.10.0-28-cloud-amd64-x86_64-with-glibc2.31",
|
||||
"gpu_count": 2,
|
||||
"gpu_type": "NVIDIA L4",
|
||||
"gpu_memory_per_device": 23580639232,
|
||||
"source": "DOCKER"
|
||||
}
|
||||
|
||||
Previewing collected data
|
||||
-------------------------
|
||||
|
||||
If you enable **local logging**, usage messages are appended to your specified log file. To inspect the most recent entries:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
tail ~/.config/lmcache/usage.log
|
||||
|
||||
Configuration & Opt-out
|
||||
-----------------------
|
||||
|
||||
By default, usage tracking is **enabled**. Any one of the following opt-outs
|
||||
disables all usage stats collection:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# LMCache-specific opt-out
|
||||
export LMCACHE_TRACK_USAGE=false
|
||||
|
||||
# The cross-tool "do not track" convention (1/true/yes)
|
||||
export DO_NOT_TRACK=1
|
||||
|
||||
When tracking is disabled, ``InitializeUsageContext`` will return ``None`` and
|
||||
no data will be sent or logged, and no state files (such as ``machine_id``)
|
||||
will be created.
|
||||
|
||||
Reference
|
||||
~~~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 34 14 52
|
||||
|
||||
* - Environment variable / file
|
||||
- Default
|
||||
- Effect
|
||||
* - ``LMCACHE_TRACK_USAGE``
|
||||
- unset
|
||||
- Set to ``false`` to disable all usage stats collection.
|
||||
* - ``DO_NOT_TRACK``
|
||||
- unset
|
||||
- Set to ``1``/``true``/``yes`` to disable collection (cross-tool
|
||||
convention).
|
||||
* - ``LMCACHE_USAGE_TRACK_URL``
|
||||
- ``http://stats.lmcache.ai:8080``
|
||||
- Override the stats server endpoint (e.g. for a private sink).
|
||||
* - ``LMCACHE_USAGE_TRACK_INTERVAL``
|
||||
- ``600``
|
||||
- Seconds between continuous-message flushes.
|
||||
* - ``~/.config/lmcache/machine_id``
|
||||
- created on first send
|
||||
- Holds the random anonymous machine UUID. Delete it to rotate the
|
||||
identifier.
|
||||
|
||||
Local logging
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
If you would like to log to a file in addition to (or instead of) sending data to the server, pass a local-log path when initializing:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from lmcache.usage_telemetry import InitializeUsageContext
|
||||
|
||||
usage_ctx = InitializeUsageContext(
|
||||
config=engine_config,
|
||||
metadata=engine_metadata,
|
||||
local_log="~/.config/lmcache/usage.log"
|
||||
)
|
||||
|
||||
Omitting the ``local_log`` argument (or passing ``None``) disables local file logging.
|
||||
Reference in New Issue
Block a user