chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
.. _internal_api_server:
|
||||
|
||||
Internal API Server
|
||||
===================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/http_api`.
|
||||
|
||||
|
||||
The ``internal_api_server`` provides HTTP APIs for managing and inspecting
|
||||
the LMCache engine at runtime. APIs are organized into three categories:
|
||||
|
||||
- **Common APIs** — Available across all components (scheduler, worker, controller).
|
||||
- **vLLM / Inference APIs** — Specific to vLLM inference workers.
|
||||
- **Controller APIs** — Specific to the LMCache Controller.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
common_apis
|
||||
vllm_apis
|
||||
controller_apis
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The following parameters can be configured in the YAML file:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Enable/disable the internal API server
|
||||
internal_api_server_enabled: True
|
||||
# Base port for the API server
|
||||
# actual_port = internal_api_server_port_start + index
|
||||
# Scheduler → 6999 + 0 = 6999
|
||||
# Worker 0 → 6999 + 1 = 7000
|
||||
internal_api_server_port_start: 6999
|
||||
# List of scheduler/worker indices: 0 for scheduler, 1 for worker 0, 2 for worker 1, etc.
|
||||
internal_api_server_include_index_list: [0, 1]
|
||||
# Socket path prefix for the API server. If configured, the server will use a Unix socket instead of listening on a port.
|
||||
internal_api_server_socket_path_prefix: "/tmp/lmcache_internal_api_server/socket"
|
||||
|
||||
# Actual socket files will be:
|
||||
# /tmp/lmcache_internal_api_server/socket_6999 (scheduler)
|
||||
# /tmp/lmcache_internal_api_server/socket_7000 (worker 0)
|
||||
|
||||
|
||||
Port Assignment
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
The port for each component is computed as:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
actual_port = internal_api_server_port_start + port_offset
|
||||
|
||||
Where ``port_offset`` is:
|
||||
|
||||
- ``0`` for the Scheduler
|
||||
- ``1 + worker_id`` for Workers (e.g., Worker 0 → offset 1, Worker 1 → offset 2)
|
||||
|
||||
|
||||
API Category & Route Discovery
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The server uses ``APIRegistry`` to automatically discover and register
|
||||
API endpoint modules. Any file named ``*_api.py`` under
|
||||
``lmcache/v1/internal_api_server/{common,vllm,controller}/`` that
|
||||
exports a ``router = APIRouter()`` will be automatically included.
|
||||
|
||||
|
||||
Extending the Server
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To add a new API endpoint:
|
||||
|
||||
1. Create a new file in the appropriate category directory
|
||||
(``common/``, ``vllm/``, or ``controller/``).
|
||||
2. Name the file with ``_api.py`` suffix (e.g., ``my_feature_api.py``).
|
||||
3. Define ``router = APIRouter()`` and add your endpoints.
|
||||
|
||||
The endpoint will be automatically discovered and registered on the
|
||||
next server startup.
|
||||
Reference in New Issue
Block a user