ray-project--ray/doc/source/serve/llm/quick-start.md

(quick-start)=
# Quickstart

## Prerequisites

```bash
pip install "ray[llm]"
```

Before you start:

- **GPU**: most models need at least one GPU. The examples below use small Qwen models that fit on a single A10G or L4. Set `accelerator_type` to a GPU available in your cluster.
- **Gated models**: to pull gated weights (for example, Llama) from the Hugging Face Hub, set `HF_TOKEN` in the deployment's `runtime_env`. See {doc}`Deployment initialization <user-guides/deployment-initialization>`.

For a full description of every configuration field used below, see the {doc}`Configuration reference <user-guides/configuration>`.

## Deployment through OpenAiIngress

You can deploy LLM models using either the builder pattern or bind pattern.

::::{tab-set}

:::{tab-item} Builder Pattern
:sync: builder

```{literalinclude} ../../llm/doc_code/serve/qwen/qwen_example.py
:language: python
:start-after: __qwen_example_start__
:end-before: __qwen_example_end__
```
:::

:::{tab-item} Bind Pattern
:sync: bind

```python
from ray import serve
from ray.serve.llm import LLMConfig
from ray.serve.llm.deployment import LLMServer
from ray.serve.llm.ingress import OpenAiIngress, make_fastapi_ingress

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-0.5b",
        model_source="Qwen/Qwen2.5-0.5B-Instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1, max_replicas=2,
        )
    ),
    # Pass the desired accelerator type (e.g. A10G, L4, etc.)
    accelerator_type="A10G",
    # You can customize the engine arguments (e.g. vLLM engine kwargs)
    engine_kwargs=dict(
        tensor_parallel_size=2,
    ),
)

# Deploy the application
server_options = LLMServer.get_deployment_options(llm_config)
server_deployment = serve.deployment(LLMServer).options(
    **server_options).bind(llm_config)

ingress_options = OpenAiIngress.get_deployment_options(
    llm_configs=[llm_config])
ingress_cls = make_fastapi_ingress(OpenAiIngress)
ingress_deployment = serve.deployment(ingress_cls).options(
    **ingress_options).bind([server_deployment])

serve.run(ingress_deployment, blocking=True)
```
:::

::::

You can query the deployed models with either cURL or the OpenAI Python client:

::::{tab-set}

:::{tab-item} cURL
:sync: curl

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer fake-key" \
     -d '{
           "model": "qwen-0.5b",
           "messages": [{"role": "user", "content": "Hello!"}]
         }'
```
:::

:::{tab-item} Python
:sync: python

```python
from openai import OpenAI

# Initialize client
client = OpenAI(base_url="http://localhost:8000/v1", api_key="fake-key")

# Basic chat completion with streaming
response = client.chat.completions.create(
    model="qwen-0.5b",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)
```
:::

::::


For deploying multiple models, you can pass a list of {class}`LLMConfig <ray.serve.llm.LLMConfig>` objects to the {class}`OpenAiIngress <ray.serve.llm.ingress.OpenAiIngress>` deployment:

::::{tab-set}

:::{tab-item} Builder Pattern
:sync: builder

```python
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app


llm_config1 = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-0.5b",
        model_source="Qwen/Qwen2.5-0.5B-Instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1, max_replicas=2,
        )
    ),
    accelerator_type="A10G",
)

llm_config2 = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-1.5b",
        model_source="Qwen/Qwen2.5-1.5B-Instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1, max_replicas=2,
        )
    ),
    accelerator_type="A10G",
)

app = build_openai_app({"llm_configs": [llm_config1, llm_config2]})
serve.run(app, blocking=True)
```
:::

:::{tab-item} Bind Pattern
:sync: bind

```python
from ray import serve
from ray.serve.llm import LLMConfig
from ray.serve.llm.deployment import LLMServer
from ray.serve.llm.ingress import OpenAiIngress, make_fastapi_ingress

llm_config1 = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-0.5b",
        model_source="Qwen/Qwen2.5-0.5B-Instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1, max_replicas=2,
        )
    ),
    accelerator_type="A10G",
)

llm_config2 = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-1.5b",
        model_source="Qwen/Qwen2.5-1.5B-Instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1, max_replicas=2,
        )
    ),
    accelerator_type="A10G",
)

# deployment #1
server_options1 = LLMServer.get_deployment_options(llm_config1)
server_deployment1 = serve.deployment(LLMServer).options(
    **server_options1).bind(llm_config1)

# deployment #2
server_options2 = LLMServer.get_deployment_options(llm_config2)
server_deployment2 = serve.deployment(LLMServer).options(
    **server_options2).bind(llm_config2)

# ingress
ingress_options = OpenAiIngress.get_deployment_options(
    llm_configs=[llm_config1, llm_config2])
ingress_cls = make_fastapi_ingress(OpenAiIngress)
ingress_deployment = serve.deployment(ingress_cls).options(
    **ingress_options).bind([server_deployment1, server_deployment2])

# run
serve.run(ingress_deployment, blocking=True)
```
:::

::::

## Production deployment

For production deployments, Ray Serve LLM provides utilities for config-driven deployments. You can specify your deployment configuration with YAML files:

::::{tab-set}

:::{tab-item} Inline Config
:sync: inline

```{literalinclude} ../../llm/doc_code/serve/qwen/llm_config_example.yaml
:language: yaml
```
:::

:::{tab-item} Standalone Config
:sync: standalone

```yaml
# config.yaml
applications:
- args:
    llm_configs:
        - models/qwen-0.5b.yaml
        - models/qwen-1.5b.yaml
  import_path: ray.serve.llm:build_openai_app
  name: llm_app
  route_prefix: "/"
```

```yaml
# models/qwen-0.5b.yaml
model_loading_config:
  model_id: qwen-0.5b
  model_source: Qwen/Qwen2.5-0.5B-Instruct
accelerator_type: A10G
deployment_config:
  autoscaling_config:
    min_replicas: 1
    max_replicas: 2
```

```yaml
# models/qwen-1.5b.yaml
model_loading_config:
  model_id: qwen-1.5b
  model_source: Qwen/Qwen2.5-1.5B-Instruct
accelerator_type: A10G
deployment_config:
  autoscaling_config:
    min_replicas: 1
    max_replicas: 2
```
:::

::::

To deploy with either configuration file:

```bash
serve run config.yaml
```

For monitoring and observability, see {doc}`Observability <user-guides/observability>`.

## Next steps

Once you can deploy and query a model, the {doc}`User guides <user-guides/index>` cover the next steps:

- **Configure the deployment**: every field is documented in the {doc}`Configuration reference <user-guides/configuration>`.
- **Scale across GPUs and nodes**: {doc}`Cross-node parallelism <user-guides/cross-node-parallelism>` distributes a model with tensor and pipeline parallelism. {doc}`Data parallel attention <user-guides/data-parallel-attention>` raises throughput by replicating the model.
- **Tune latency and throughput**: {doc}`Prefill/decode disaggregation <user-guides/prefill-decode>`, {doc}`KV cache offloading <user-guides/kv-cache-offloading>`, and {doc}`Prefix-aware routing <user-guides/prefix-aware-routing>`.
- **Serve LoRA adapters**: {doc}`Multi-LoRA deployment <user-guides/multi-lora>`.
- **Monitor in production**: {doc}`Observability and monitoring <user-guides/observability>`.

To understand how these pieces fit together, see the {doc}`Architecture <architecture/index>` docs.