ray-project--ray/doc/source/serve/llm/user-guides/index.md

User guides

How-to guides for deploying, scaling, and operating Ray Serve LLM. If you are new, start with the {doc}Quickstart <../quick-start>, then come back here to go deeper.

Configure and deploy

• {doc}Configuration reference <configuration>: every LLMConfig field, from model loading and engine kwargs to accelerators, placement, and deployment options.
• {doc}Deployment initialization <deployment-initialization>: speed up model loading and replica startup with caching, streaming load formats, and initialization callbacks.
• {doc}Multi-LoRA deployment <multi-lora>: serve many LoRA adapters on a shared base model with runtime switching and an LRU cache.

Scale across GPUs and nodes

• {doc}Cross-node parallelism <cross-node-parallelism>: distribute a model across GPUs and nodes with tensor and pipeline parallelism and placement groups.
• {doc}Data parallel attention <data-parallel-attention>: replicate the model into coordinated data-parallel groups to raise throughput, especially for MoE models.
• {doc}Fractional GPU serving <fractional-gpu>: pack multiple small-model replicas onto a single GPU.

Optimize latency and throughput

• {doc}Prefill/decode disaggregation <prefill-decode>: split prompt processing and token generation onto separate replicas to tune each independently.
• {doc}KV cache offloading <kv-cache-offloading>: extend KV cache capacity with LMCache and tiered storage backends.
• {doc}Prefix-aware routing <prefix-aware-routing>: route requests to replicas that already hold a matching prefix to maximize cache hits.
• {doc}Direct streaming <direct-streaming>: bypass the ingress when streaming tokens to cut per-token latency.

Choose an engine

• {doc}vLLM compatibility <vllm-compatibility>: use vLLM features such as embeddings, structured outputs, vision, and reasoning through Ray Serve LLM.
• {doc}Custom vLLM models <custom-vllm>: serve an out-of-tree architecture with a vLLM plugin, using a Qwen3 reward model as the example.
• {doc}SGLang integration <sglang>: run SGLang as the inference engine instead of vLLM.

Operate in production

• {doc}Observability and monitoring <observability>: engine and request metrics, Grafana dashboards, and Prometheus integration.

:hidden:
:maxdepth: 1

Configuration reference <configuration>
Deployment initialization <deployment-initialization>
Multi-LoRA deployment <multi-lora>
Cross-node parallelism <cross-node-parallelism>
Data parallel attention <data-parallel-attention>
Fractional GPU serving <fractional-gpu>
Prefill/decode disaggregation <prefill-decode>
KV cache offloading <kv-cache-offloading>
Prefix-aware routing <prefix-aware-routing>
Direct streaming <direct-streaming>
vLLM compatibility <vllm-compatibility>
Custom vLLM models <custom-vllm>
SGLang integration <sglang>
Observability and monitoring <observability>