TensorRT-LLM integration

Adapter shape

lmcache/integration/tensorrt_llm/
├── __init__.py             # Optional-import surface
├── utils.py                # ENGINE_NAME, lmcache_get_config,
│                           # create_trtllm_metadata
├── tensorrt_adapter.py     # In-process — engine in TRT-LLM process
└── tensorrt_mp_adapter.py  # Multi-process — engine in standalone server

Both adapters subclass TRT-LLM's KvCacheConnectorScheduler and KvCacheConnectorWorker. The TRT-LLM imports are guarded at module level only via the package's __init__; nothing in core LMCache imports the adapter modules. This keeps pip install lmcache unaffected when TRT-LLM is absent.

The TRT-LLM connector preset registry (PR NVIDIA/TensorRT-LLM#12626) maps:

Preset	Module	Scheduler	Worker
`lmcache`	`lmcache.integration.tensorrt_llm.tensorrt_adapter`	`LMCacheKvConnectorScheduler`	`LMCacheKvConnectorWorker`
`lmcache-mp`	`lmcache.integration.tensorrt_llm.tensorrt_mp_adapter`	`LMCacheMPKvConnectorScheduler`	`LMCacheMPKvConnectorWorker`

Lifecycle

Stage	TRT-LLM hook	LMCache call
Init	`worker.register_kv_caches(kv_cache_tensor)`	Build engine via `_get_or_create_engine`; call `gpu_connector.register_kv_caches(kv_cache_tensor)`
Before scheduling	`scheduler.get_num_new_matched_tokens(req, num_computed)`	`engine.lookup(tokens)` (in-process) or `LOOKUP` + `QUERY_PREFETCH_STATUS` (MP)
Pre-forward	`scheduler.build_connector_meta(scheduler_output)`	`LMCacheConnectorMetadata(loads=..., saves=...)`
Forward	`worker.start_load_kv(stream)`	`engine.retrieve(tokens, block_ids)`
Forward	`worker.wait_for_save(stream)`	`engine.store(tokens, block_ids)`

In-process vs MP

The two modes share the lifecycle but differ in where state lives.

Aspect	In-process (`lmcache`)	Multi-process (`lmcache-mp`)
LMCache engine	Singleton inside the TRT-LLM process	Standalone ZMQ server
Tensor sharing	Direct (same process)	`RawCudaIPCWrapper` (cudaIpc + cupy DLPack)
Lookup	`engine.lookup(tokens)` returns chunk count	`LOOKUP` enqueues prefetch; `QUERY_PREFETCH_STATUS` reads result keyed by `request_id`
Configuration	`LMCACHE_CONFIG_FILE` env var	Same; plus `server_url` in connector config (or `LMCACHE_SERVER_URL` env)
Failure mode	One process crash takes down both	Engine survives TRT-LLM crash; multiple TRT-LLM instances can share cache
Setup cost	None	Run `python -m lmcache.v1.multiprocess.server`

Why not subclass `VLLMPagedMemGPUConnectorV3`

V3's transfer path is wrong for TRT-LLM:

V3 uses the in-process kernel (multi_layer_kv_transfer) with a slot_mapping of token positions and per-layer pointers. TRT-LLM's cross-layer pool is a single base pointer and we want to transfer by block ids, not slot positions.
TRT-LLM needs the MP kernel (multi_layer_block_kv_transfer) which natively handles single-base-pointer cross-layer with shape_desc.nl walking layers internally. There is nothing to inherit.

TRTLLMGPUConnector is therefore a standalone GPUConnectorInterface implementation. It also exposes a bespoke register_kv_caches(kv_cache_tensor) method called by the worker once at init — separate from to_gpu/from_gpu. The factory in lmcache/v1/gpu_connector/__init__.py constructs it from LMCacheMetadata plus the device, and the adapter wires the pool tensor in afterwards.

Forcing real LMCache hits in tests

TRT-LLM has its own GPU block reuse. To verify LMCache contributes the hit (and not TRT-LLM's reuse), the E2E tests size TRT-LLM's pool tiny (KvCacheConfig(max_tokens=512)) while sending prompts much larger than 512 tokens. The first request fills LMCache and TRT-LLM. The second is guaranteed-evicted from TRT-LLM's pool and must come from LMCache — which the test asserts via the lmcache_cached=… new_matched=… log line on request 3.

README.md