chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,355 @@
|
||||
.. _common_apis:
|
||||
|
||||
Common APIs
|
||||
===========
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/http_api`.
|
||||
|
||||
|
||||
Common APIs are available across all components (scheduler, worker, controller).
|
||||
|
||||
.. contents:: Endpoints
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
|
||||
``GET /env`` — Environment Variables
|
||||
-------------------------------------
|
||||
|
||||
Get all environment variables of the running process.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/env``
|
||||
- **Parameters**: None
|
||||
- **Response**: ``application/json`` — JSON object of all environment variables (sorted by key).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:7000/env
|
||||
|
||||
**Example Response**:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"HOME": "/root",
|
||||
"PATH": "/usr/local/bin:/usr/bin",
|
||||
"PYTHONPATH": "/app"
|
||||
}
|
||||
|
||||
|
||||
``GET /loglevel`` — Log Level Management
|
||||
------------------------------------------
|
||||
|
||||
Get or set the log level for Python loggers. Behavior depends on query parameters:
|
||||
|
||||
- **No parameters**: List all loggers and their levels.
|
||||
- **``logger_name`` only**: Get the level of the specified logger.
|
||||
- **``logger_name`` and ``level``**: Set the level of the specified logger (including all its handlers).
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/loglevel``
|
||||
- **Parameters**:
|
||||
|
||||
=============== ======== ============================================
|
||||
Name Type Description
|
||||
=============== ======== ============================================
|
||||
``logger_name`` str (Optional) Logger name to query or set
|
||||
``level`` str (Optional) Log level to set (e.g. ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``)
|
||||
=============== ======== ============================================
|
||||
|
||||
- **Response**: ``text/plain``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# List all loggers
|
||||
curl http://localhost:7000/loglevel
|
||||
|
||||
# Get a specific logger level
|
||||
curl "http://localhost:7000/loglevel?logger_name=lmcache.v1.cache_engine"
|
||||
|
||||
# Set a specific logger level
|
||||
curl "http://localhost:7000/loglevel?logger_name=lmcache.v1.cache_engine&level=DEBUG"
|
||||
|
||||
**Example Response** (list all):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
=== Loggers and Levels ===
|
||||
lmcache.v1.cache_engine: WARNING
|
||||
lmcache.v1.storage_backend: INFO
|
||||
|
||||
**Example Response** (get):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
lmcache.v1.cache_engine: WARNING
|
||||
|
||||
**Example Response** (set):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Set lmcache.v1.cache_engine level to DEBUG (including all handlers)
|
||||
|
||||
**Error Response** (invalid level, HTTP 400):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Invalid log level: INVALID_LEVEL
|
||||
|
||||
|
||||
``GET /metrics`` — Prometheus Metrics
|
||||
--------------------------------------
|
||||
|
||||
Get Prometheus metrics data in the standard exposition format.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/metrics``
|
||||
- **Parameters**: None
|
||||
- **Response**: ``text/plain`` — Prometheus text-based exposition format.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:7000/metrics
|
||||
|
||||
|
||||
``POST /metrics/reset`` — Reset Prometheus Metrics
|
||||
----------------------------------------------------
|
||||
|
||||
Reset all Prometheus metrics to their initial state.
|
||||
|
||||
- **Method**: ``POST``
|
||||
- **Path**: ``/metrics/reset``
|
||||
- **Parameters**: None
|
||||
- **Response**: ``text/plain`` — ``"ok"`` on success.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:7000/metrics/reset
|
||||
|
||||
|
||||
``GET /threads`` — Thread Information
|
||||
--------------------------------------
|
||||
|
||||
Get information about active threads with optional filtering.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/threads``
|
||||
- **Parameters**:
|
||||
|
||||
============== ======= ===========================================
|
||||
Name Type Description
|
||||
============== ======= ===========================================
|
||||
``name`` str (Optional) Filter by thread name (fuzzy match, case-insensitive)
|
||||
``thread_id`` int (Optional) Filter by thread ID
|
||||
============== ======= ===========================================
|
||||
|
||||
- **Response**: ``text/plain`` — Thread info with stack traces and summary.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Get all threads
|
||||
curl http://localhost:7000/threads
|
||||
|
||||
# Filter by name
|
||||
curl "http://localhost:7000/threads?name=api-server"
|
||||
|
||||
# Filter by thread ID
|
||||
curl "http://localhost:7000/threads?thread_id=12345"
|
||||
|
||||
|
||||
``GET /periodic-threads`` — Periodic Thread Status
|
||||
----------------------------------------------------
|
||||
|
||||
Get information about registered periodic threads.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/periodic-threads``
|
||||
- **Parameters**:
|
||||
|
||||
================ ======= =============================================
|
||||
Name Type Description
|
||||
================ ======= =============================================
|
||||
``level`` str (Optional) Filter by thread level: ``critical``, ``high``, ``medium``, ``low``
|
||||
``running_only`` bool (Optional) Only show running threads (default: ``false``)
|
||||
``active_only`` bool (Optional) Only show active threads (default: ``false``)
|
||||
================ ======= =============================================
|
||||
|
||||
- **Response**: ``application/json``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Get all periodic threads
|
||||
curl http://localhost:7000/periodic-threads
|
||||
|
||||
# Filter by level
|
||||
curl "http://localhost:7000/periodic-threads?level=critical"
|
||||
|
||||
# Only running threads
|
||||
curl "http://localhost:7000/periodic-threads?running_only=true"
|
||||
|
||||
**Example Response**:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"summary": {
|
||||
"total_count": 5,
|
||||
"running_count": 3,
|
||||
"active_count": 3,
|
||||
"by_level": {
|
||||
"critical": {"total": 1, "running": 1, "active": 1},
|
||||
"high": {"total": 2, "running": 1, "active": 1}
|
||||
}
|
||||
},
|
||||
"threads": [
|
||||
{
|
||||
"name": "heartbeat",
|
||||
"level": "critical",
|
||||
"is_running": true,
|
||||
"is_active": true,
|
||||
"last_run_time": "2025-01-01T00:00:00",
|
||||
"interval": 10.0
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
**Error Response** (invalid level, HTTP 400):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"error": "Invalid level: unknown. Valid values: critical, high, medium, low"
|
||||
}
|
||||
|
||||
|
||||
``GET /periodic-threads/{thread_name}`` — Single Periodic Thread
|
||||
------------------------------------------------------------------
|
||||
|
||||
Get detailed information about a specific periodic thread by name.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/periodic-threads/{thread_name}``
|
||||
- **Path Parameters**:
|
||||
|
||||
================ ======= =============================================
|
||||
Name Type Description
|
||||
================ ======= =============================================
|
||||
``thread_name`` str Name of the periodic thread
|
||||
================ ======= =============================================
|
||||
|
||||
- **Response**: ``application/json``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:7000/periodic-threads/heartbeat
|
||||
|
||||
**Error Response** (not found, HTTP 404):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"error": "Thread not found: heartbeat"
|
||||
}
|
||||
|
||||
|
||||
``GET /periodic-threads-health`` — Periodic Thread Health Check
|
||||
-----------------------------------------------------------------
|
||||
|
||||
Quick health check for periodic threads. Returns healthy status if all
|
||||
``critical`` and ``high`` level threads are active.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/periodic-threads-health``
|
||||
- **Parameters**: None
|
||||
- **Response**: ``application/json``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:7000/periodic-threads-health
|
||||
|
||||
**Example Response** (healthy):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"healthy": true,
|
||||
"unhealthy_count": 0,
|
||||
"unhealthy_threads": []
|
||||
}
|
||||
|
||||
**Example Response** (unhealthy):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"healthy": false,
|
||||
"unhealthy_count": 1,
|
||||
"unhealthy_threads": [
|
||||
{
|
||||
"name": "heartbeat",
|
||||
"level": "critical",
|
||||
"last_run_ago": 120.5,
|
||||
"interval": 10.0
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
``POST /run_script`` — Run Script
|
||||
-----------------------------------
|
||||
|
||||
Upload and execute a Python script in a restricted sandbox environment.
|
||||
The script has access to ``app`` (the FastAPI application instance) and
|
||||
a limited set of builtins. Import is restricted to modules configured
|
||||
in ``script_allowed_imports``.
|
||||
|
||||
- **Method**: ``POST``
|
||||
- **Path**: ``/run_script``
|
||||
- **Content-Type**: ``multipart/form-data``
|
||||
- **Parameters**:
|
||||
|
||||
============ ======== =============================================
|
||||
Name Type Description
|
||||
============ ======== =============================================
|
||||
``script`` file Python script file to execute
|
||||
============ ======== =============================================
|
||||
|
||||
- **Response**: ``text/plain`` — The ``result`` variable from the script, or
|
||||
``"Script executed successfully"`` if no ``result`` is set.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:7000/run_script \
|
||||
-F "script=@/path/to/scratch.py"
|
||||
|
||||
**Example Script** (``scratch.py``):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
lmcache_engine = app.state.lmcache_adapter.lmcache_engine
|
||||
|
||||
result = {
|
||||
"is_first_rank": lmcache_engine.metadata.is_first_rank(),
|
||||
"model_version": lmcache_engine.metadata.kv_shape,
|
||||
}
|
||||
|
||||
**Example Response**:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
{'is_first_rank': True, 'model_version': (27, 1, 64, 1, 576)}
|
||||
|
||||
**Error Response** (no script, HTTP 400):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
No script file provided
|
||||
|
||||
**Error Response** (execution error, HTTP 500):
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Error executing script: Import of 'os' is not allowed
|
||||
@@ -0,0 +1,201 @@
|
||||
.. _controller_apis:
|
||||
|
||||
Controller APIs
|
||||
===============
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/http_api`.
|
||||
|
||||
|
||||
These APIs are specific to the LMCache Controller component. They provide
|
||||
visibility into registered instances, workers, and key statistics.
|
||||
|
||||
.. contents:: Endpoints
|
||||
:local:
|
||||
:depth: 2
|
||||
|
||||
|
||||
``GET /controller/key-stats`` — Key Statistics
|
||||
------------------------------------------------
|
||||
|
||||
Get key statistics across all instances and workers.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/controller/key-stats``
|
||||
- **Parameters**: None
|
||||
- **Response**: ``application/json``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:6999/controller/key-stats
|
||||
|
||||
**Example Response** (HTTP 200):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"total_key_count": 1500,
|
||||
"total_instance_count": 2,
|
||||
"total_worker_count": 4,
|
||||
"instances": [
|
||||
{
|
||||
"instance_id": "instance_001",
|
||||
"key_count": 800,
|
||||
"worker_count": 2
|
||||
},
|
||||
{
|
||||
"instance_id": "instance_002",
|
||||
"key_count": 700,
|
||||
"worker_count": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
**Error Response** (controller not available, HTTP 503):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"detail": "Controller manager not available"
|
||||
}
|
||||
|
||||
**Response Schema**:
|
||||
|
||||
========================= ======= =============================================
|
||||
Field Type Description
|
||||
========================= ======= =============================================
|
||||
``total_key_count`` int Total number of KV keys across all instances
|
||||
``total_instance_count`` int Total number of registered instances
|
||||
``total_worker_count`` int Total number of workers across all instances
|
||||
``instances`` list Per-instance breakdown (see below)
|
||||
========================= ======= =============================================
|
||||
|
||||
**Instance Schema**:
|
||||
|
||||
================= ======= =============================================
|
||||
Field Type Description
|
||||
================= ======= =============================================
|
||||
``instance_id`` str Unique identifier of the instance
|
||||
``key_count`` int Number of KV keys held by this instance
|
||||
``worker_count`` int Number of workers in this instance
|
||||
================= ======= =============================================
|
||||
|
||||
|
||||
``GET /controller/workers`` — Worker Information
|
||||
--------------------------------------------------
|
||||
|
||||
Get worker information with flexible query parameters. Behavior depends
|
||||
on the combination of parameters:
|
||||
|
||||
- **No parameters**: List all registered workers across all instances.
|
||||
- **``instance_id`` only**: List all workers for a specific instance.
|
||||
- **``instance_id`` and ``worker_id``**: Get detailed info about a specific worker.
|
||||
|
||||
- **Method**: ``GET``
|
||||
- **Path**: ``/controller/workers``
|
||||
- **Parameters**:
|
||||
|
||||
================ ======= =============================================
|
||||
Name Type Description
|
||||
================ ======= =============================================
|
||||
``instance_id`` str (Optional) Instance ID to filter workers
|
||||
``worker_id`` int (Optional) Worker ID for specific worker details (requires ``instance_id``)
|
||||
================ ======= =============================================
|
||||
|
||||
- **Response**: ``application/json``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# List all workers
|
||||
curl http://localhost:6999/controller/workers
|
||||
|
||||
# List workers for a specific instance
|
||||
curl "http://localhost:6999/controller/workers?instance_id=instance_001"
|
||||
|
||||
# Get a specific worker
|
||||
curl "http://localhost:6999/controller/workers?instance_id=instance_001&worker_id=0"
|
||||
|
||||
**Example Response** (list workers):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"workers": [
|
||||
{
|
||||
"instance_id": "instance_001",
|
||||
"worker_id": 0,
|
||||
"ip": "10.0.0.1",
|
||||
"port": 8000,
|
||||
"peer_init_url": "http://10.0.0.1:8000/init",
|
||||
"registration_time": 1706745600.0,
|
||||
"last_heartbeat_time": 1706745660.0,
|
||||
"key_count": 400
|
||||
},
|
||||
{
|
||||
"instance_id": "instance_001",
|
||||
"worker_id": 1,
|
||||
"ip": "10.0.0.2",
|
||||
"port": 8001,
|
||||
"peer_init_url": "http://10.0.0.2:8001/init",
|
||||
"registration_time": 1706745600.0,
|
||||
"last_heartbeat_time": 1706745660.0,
|
||||
"key_count": 400
|
||||
}
|
||||
],
|
||||
"total_count": 2
|
||||
}
|
||||
|
||||
**Example Response** (single worker):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"instance_id": "instance_001",
|
||||
"worker_id": 0,
|
||||
"ip": "10.0.0.1",
|
||||
"port": 8000,
|
||||
"peer_init_url": "http://10.0.0.1:8000/init",
|
||||
"registration_time": 1706745600.0,
|
||||
"last_heartbeat_time": 1706745660.0,
|
||||
"key_count": 400
|
||||
}
|
||||
|
||||
**Error Response** (worker not found, HTTP 404):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"detail": "Worker (instance_001, 99) not found"
|
||||
}
|
||||
|
||||
**Error Response** (instance not found, HTTP 404):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"detail": "No workers found for instance unknown_instance"
|
||||
}
|
||||
|
||||
**Error Response** (controller not available, HTTP 503):
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"detail": "Controller manager not available"
|
||||
}
|
||||
|
||||
**Worker Response Schema**:
|
||||
|
||||
========================== ======= =============================================
|
||||
Field Type Description
|
||||
========================== ======= =============================================
|
||||
``instance_id`` str Instance this worker belongs to
|
||||
``worker_id`` int Worker index within the instance
|
||||
``ip`` str Worker IP address
|
||||
``port`` int Worker port number
|
||||
``peer_init_url`` str (Optional) Peer initialization URL
|
||||
``registration_time`` float Unix timestamp of worker registration
|
||||
``last_heartbeat_time`` float Unix timestamp of last heartbeat
|
||||
``key_count`` int Number of KV keys held by this worker
|
||||
========================== ======= =============================================
|
||||
@@ -0,0 +1,181 @@
|
||||
.. _dynamic_backend_management:
|
||||
|
||||
Dynamic Backend Management
|
||||
==========================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/http_api`.
|
||||
|
||||
|
||||
LMCache provides a set of internal API endpoints that allow you to **list**,
|
||||
**close**, and **create** storage backends at runtime without restarting the
|
||||
serving engine. This is useful when you need to switch between different
|
||||
storage configurations on the fly — for example, migrating from a
|
||||
``LocalDiskBackend`` to a ``GdsBackend``, or changing the remote connector
|
||||
from a filesystem connector to Redis.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The workflow for dynamically switching a storage backend is:
|
||||
|
||||
1. **Close** the backend you want to replace.
|
||||
2. **Update** the relevant configuration via the ``POST /conf`` API.
|
||||
3. **Create** new backends — only backends that are not already present
|
||||
will be created.
|
||||
|
||||
Any backend that was **not** closed will be skipped during creation,
|
||||
so the operation is safe and idempotent.
|
||||
|
||||
API Endpoints
|
||||
-------------
|
||||
|
||||
``GET /backends``
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
List all active storage backends.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:7000/backends
|
||||
|
||||
Response:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"LocalCPUBackend": "LocalCPUBackend",
|
||||
"RemoteBackend": "RemoteBackend"
|
||||
}
|
||||
|
||||
``DELETE /backends/{backend_name}``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Close and remove a specific storage backend. After this call the backend
|
||||
is fully shut down and removed from the internal dictionary, ensuring no
|
||||
stale references remain.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X DELETE http://localhost:7000/backends/RemoteBackend
|
||||
|
||||
Response:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"status": "success",
|
||||
"message": "Backend RemoteBackend closed",
|
||||
"backends": {
|
||||
"LocalCPUBackend": "LocalCPUBackend"
|
||||
}
|
||||
}
|
||||
|
||||
``POST /backends``
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Create new storage backends based on the current ``LMCacheEngineConfig``.
|
||||
Existing backends are skipped.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:7000/backends
|
||||
|
||||
Response:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"status": "success",
|
||||
"created": {
|
||||
"RemoteBackend": "RemoteBackend"
|
||||
},
|
||||
"backends": {
|
||||
"LocalCPUBackend": "LocalCPUBackend",
|
||||
"RemoteBackend": "RemoteBackend"
|
||||
}
|
||||
}
|
||||
|
||||
Use-Case Examples
|
||||
-----------------
|
||||
|
||||
Switching from ``LocalDiskBackend`` to ``GdsBackend``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If you originally configured a local-disk backend and want to migrate to
|
||||
NVIDIA GPUDirect Storage (GDS) at runtime:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# 1. Close the old disk backend
|
||||
curl -X DELETE http://localhost:7000/backends/LocalDiskBackend
|
||||
|
||||
# 2. Disable local_disk and set the GDS path
|
||||
curl -X POST http://localhost:7000/conf \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"local_disk": false,
|
||||
"gds_path": "/mnt/nvme/lmcache_gds"
|
||||
}'
|
||||
|
||||
# 3. Create the new GDS backend
|
||||
curl -X POST http://localhost:7000/backends
|
||||
|
||||
# 4. Verify the new backend list
|
||||
curl http://localhost:7000/backends
|
||||
|
||||
Switching ``RemoteBackend`` connector (FS → Redis)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To change the remote connector type without restarting:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# 1. Close the old remote backend
|
||||
curl -X DELETE http://localhost:7000/backends/RemoteBackend
|
||||
|
||||
# 2. Update the remote URL to point to Redis
|
||||
curl -X POST http://localhost:7000/conf \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"remote_url": "redis://redis-host:6379"
|
||||
}'
|
||||
|
||||
# 3. Create the new remote backend with Redis connector
|
||||
curl -X POST http://localhost:7000/backends
|
||||
|
||||
# 4. Verify
|
||||
curl http://localhost:7000/backends
|
||||
|
||||
Replacing only one backend while keeping others
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If only the ``RemoteBackend`` needs to be updated but the
|
||||
``LocalCPUBackend`` should stay untouched:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Close only the remote backend
|
||||
curl -X DELETE http://localhost:7000/backends/RemoteBackend
|
||||
|
||||
# Update config
|
||||
curl -X POST http://localhost:7000/conf \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"remote_url": "redis://new-redis:6379"}'
|
||||
|
||||
# Create — LocalCPUBackend is skipped (already present)
|
||||
curl -X POST http://localhost:7000/backends
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
- Closing the ``LocalCPUBackend`` is possible but should be done with
|
||||
caution since many other backends rely on it as an intermediate buffer.
|
||||
- The ``POST /backends`` endpoint calls the same ``CreateStorageBackends``
|
||||
factory that is used during engine initialization, so all backend types
|
||||
(local CPU, local disk, GDS, remote, P2P, plugins, etc.) are
|
||||
supported.
|
||||
- After creating backends, the ``StorageManager`` automatically refreshes
|
||||
its internal references (``non_allocator_backends``,
|
||||
``local_cpu_backend``, etc.).
|
||||
@@ -0,0 +1,85 @@
|
||||
.. _internal_api_server:
|
||||
|
||||
Internal API Server
|
||||
===================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/http_api`.
|
||||
|
||||
|
||||
The ``internal_api_server`` provides HTTP APIs for managing and inspecting
|
||||
the LMCache engine at runtime. APIs are organized into three categories:
|
||||
|
||||
- **Common APIs** — Available across all components (scheduler, worker, controller).
|
||||
- **vLLM / Inference APIs** — Specific to vLLM inference workers.
|
||||
- **Controller APIs** — Specific to the LMCache Controller.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
common_apis
|
||||
vllm_apis
|
||||
controller_apis
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The following parameters can be configured in the YAML file:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Enable/disable the internal API server
|
||||
internal_api_server_enabled: True
|
||||
# Base port for the API server
|
||||
# actual_port = internal_api_server_port_start + index
|
||||
# Scheduler → 6999 + 0 = 6999
|
||||
# Worker 0 → 6999 + 1 = 7000
|
||||
internal_api_server_port_start: 6999
|
||||
# List of scheduler/worker indices: 0 for scheduler, 1 for worker 0, 2 for worker 1, etc.
|
||||
internal_api_server_include_index_list: [0, 1]
|
||||
# Socket path prefix for the API server. If configured, the server will use a Unix socket instead of listening on a port.
|
||||
internal_api_server_socket_path_prefix: "/tmp/lmcache_internal_api_server/socket"
|
||||
|
||||
# Actual socket files will be:
|
||||
# /tmp/lmcache_internal_api_server/socket_6999 (scheduler)
|
||||
# /tmp/lmcache_internal_api_server/socket_7000 (worker 0)
|
||||
|
||||
|
||||
Port Assignment
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
The port for each component is computed as:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
actual_port = internal_api_server_port_start + port_offset
|
||||
|
||||
Where ``port_offset`` is:
|
||||
|
||||
- ``0`` for the Scheduler
|
||||
- ``1 + worker_id`` for Workers (e.g., Worker 0 → offset 1, Worker 1 → offset 2)
|
||||
|
||||
|
||||
API Category & Route Discovery
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The server uses ``APIRegistry`` to automatically discover and register
|
||||
API endpoint modules. Any file named ``*_api.py`` under
|
||||
``lmcache/v1/internal_api_server/{common,vllm,controller}/`` that
|
||||
exports a ``router = APIRouter()`` will be automatically included.
|
||||
|
||||
|
||||
Extending the Server
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To add a new API endpoint:
|
||||
|
||||
1. Create a new file in the appropriate category directory
|
||||
(``common/``, ``vllm/``, or ``controller/``).
|
||||
2. Name the file with ``_api.py`` suffix (e.g., ``my_feature_api.py``).
|
||||
3. Define ``router = APIRouter()`` and add your endpoints.
|
||||
|
||||
The endpoint will be automatically discovered and registered on the
|
||||
next server startup.
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user