wehub-resource-sync
74 lines
2.3 KiB
ReStructuredText
74 lines
2.3 KiB
ReStructuredText
.. _query_worker_info:
|
|
|
|
Query Worker Info
|
|
=================================
|
|
|
|
.. warning::
|
|
|
|
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
|
|
|
|
|
The ``query_worker_info`` interface is defined as the following:
|
|
|
|
.. code-block:: python
|
|
|
|
query_worker_info(instance_id: str, worker_ids: List[int]) -> event_id: str, worker_infos: List[WorkerInfo]
|
|
|
|
The function get the info of the workers which specified by ``instance_id`` and ``worker_ids``.
|
|
The controller returns an ``event_id`` and the worker infos.
|
|
|
|
Example usage:
|
|
---------------------------------------
|
|
|
|
First, create a yaml file ``example.yaml`` to configure the lmcache instance:
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 256
|
|
local_cpu: True
|
|
max_local_cpu_size: 5
|
|
|
|
# cache controller configurations
|
|
enable_controller: True
|
|
lmcache_instance_id: "lmcache_default_instance"
|
|
controller_pull_url: "localhost:9001"
|
|
lmcache_worker_ports: 8001
|
|
|
|
# Peer identifiers
|
|
p2p_host: "localhost"
|
|
p2p_init_ports: 8200
|
|
|
|
|
|
Start the vllm/lmcache instance at port 8000:
|
|
|
|
.. code-block:: bash
|
|
|
|
CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=example.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 4096 \
|
|
--gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
|
|
|
Start the lmcache controller at port 9000 and the monitor at port 9001:
|
|
|
|
.. code-block:: bash
|
|
|
|
lmcache_controller --host localhost --port 9000 --monitor-port 9001
|
|
|
|
Send a request to controller:
|
|
|
|
.. code-block:: bash
|
|
|
|
curl -X POST http://localhost:9000/query_worker_info \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"instance_id": "lmcache_default_instance",
|
|
"worker_ids": [0]
|
|
}'
|
|
|
|
The controller responds with a message similar to:
|
|
|
|
.. code-block:: text
|
|
|
|
{"event_id": "xxx", "worker_infos": [{"instance_id": "lmcache_default_instance", "worker_id": 0, "ip": "127.0.0.1", "port": 8001, "peer_init_url": "127.0.0.1:8200", "registration_time": 123456, "last_heartbeat_time": 456789}]}
|
|
|
|
``worker_infos`` contains the queried worker information.
|
|
returned ``event_id`` can be used to query the status of the operation.
|