chore: import upstream snapshot with attribution

docs_new/docs/basic_usage/anthropic_api.mdx

@@ -0,0 +1,287 @@
+---
+title: "Anthropic-Compatible API"
+description: "Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning."
+---
+SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic
+Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a
+self-hosted SGLang server without changes. A complete reference for the API is available in the
+[Anthropic API Reference](https://docs.anthropic.com/en/api/messages).
+
+The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it.
+It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible
+endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route.
+
+This tutorial covers:
+
+- `POST /v1/messages` (non-streaming and streaming)
+- `POST /v1/messages/count_tokens`
+- Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is
+  required for good prefix-cache reuse.
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint
+is registered automatically — no extra flag is required beyond the usual server launch. The example below
+is a single-node GLM-5.2-FP8 config; see the
+[GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands
+across hardware and quantizations.
+
+```bash Command
+sglang serve \
+    --model-path zai-org/GLM-5.2-FP8 \
+    --tp 8 \
+    --speculative-algorithm EAGLE \
+    --speculative-num-steps 5 \
+    --speculative-eagle-topk 1 \
+    --speculative-num-draft-tokens 6 \
+    --reasoning-parser glm45 \
+    --tool-call-parser glm47 \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+<Note>
+- **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is
+  used here because its reasoning + tool-use output is where Claude Code integration shines, but any model
+  works.
+- **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send
+  any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when
+  the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the
+  `ANTHROPIC_DEFAULT_*_MODEL` env vars below.
+- **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning
+  content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use`
+  blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back
+  as raw text, and Claude Code cannot execute them.
+- **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it.
+</Note>
+
+## Send A Message
+
+### Non-Streaming
+
+Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends
+`/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix.
+
+```python Example
+from anthropic import Anthropic
+
+client = Anthropic(
+    base_url="http://127.0.0.1:30000",
+    api_key="EMPTY",  # SGLang does not require a real key by default
+)
+
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    messages=[{"role": "user", "content": "List 3 countries and their capitals."}],
+)
+# A reasoning model may emit a `thinking` block before the `text` block —
+# pick the text block rather than assuming content[0].
+print(next(b.text for b in message.content if b.type == "text"))
+```
+
+**Example Output:**
+```text Output
+Here are 3 countries and their capitals:
+
+1. **France** - Paris
+2. **Japan** - Tokyo
+3. **Brazil** - Brasília
+```
+
+### Streaming
+
+Set `stream=True` to receive Server-Sent Events as they are produced.
+
+```python Example
+with client.messages.stream(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    messages=[{"role": "user", "content": "Say this is a test"}],
+) as stream:
+    for text in stream.text_stream:
+        print(text, end="", flush=True)
+```
+
+**Example Output:**
+```text Output
+This is a test.
+```
+
+### System Prompt
+
+The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic
+API shape:
+
+```python Example
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    system="You are a helpful assistant that answers concisely.",
+    messages=[{"role": "user", "content": "What is the capital of France?"}],
+)
+print(next(b.text for b in message.content if b.type == "text"))
+```
+
+**Example Output:**
+```text Output
+The capital of France is Paris.
+```
+
+### Tool Use
+
+Tool definitions follow the Anthropic `tools` schema. When the server is launched with a
+`--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks:
+
+```python Example
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    tools=[
+        {
+            "name": "get_weather",
+            "description": "Get the weather for a city",
+            "input_schema": {
+                "type": "object",
+                "properties": {"city": {"type": "string"}},
+                "required": ["city"],
+            },
+        }
+    ],
+    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
+)
+print(message.stop_reason)
+print([b for b in message.content if b.type == "tool_use"])
+```
+
+**Example Output:**
+```text Output
+tool_use
+[ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})]
+```
+
+### Counting Tokens
+
+`POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a
+response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and
+multi-turn history are all accounted for.
+
+```python Example
+resp = client.messages.count_tokens(
+    model="zai-org/GLM-5.2-FP8",
+    messages=[{"role": "user", "content": "Hello, world"}],
+)
+print(resp.input_tokens)
+```
+
+**Example Output:**
+```text Output
+15
+```
+
+## Using Claude Code
+
+Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it.
+With the server already running on `:30000`, export the full set and launch `claude`:
+
+```bash Command
+export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
+export ANTHROPIC_AUTH_TOKEN="dummy"                 # required by Claude Code; any non-empty string works
+export API_TIMEOUT_MS="3000000"                     # long timeout — reasoning + 1M-context turns are slow
+export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"     # let auto-compact use the full 1M window
+export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1   # drop autoupdater/telemetry/error-reporting noise
+export CLAUDE_CODE_ATTRIBUTION_HEADER=0             # required for prefix-cache reuse — see below
+export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"    # [1m] suffix enables Claude Code's 1M-context beta
+export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"  # [1m] suffix enables Claude Code's 1M-context beta
+export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"     # [1m] suffix enables Claude Code's 1M-context beta
+claude
+```
+
+Each var matters:
+
+- **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API.
+- **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value
+  when launched without `--api-key`.
+- **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely
+  exceed the default timeout.
+- **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier.
+  SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a
+  client-side hint that enables Claude Code's 1M-context beta (without it, context is capped).
+- **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window
+  instead of the default, keeping long sessions alive.
+
+<Tip>
+Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key
+— they apply to all Claude Code sessions:
+
+```json
+{
+  "env": {
+    "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000",
+    "ANTHROPIC_AUTH_TOKEN": "dummy",
+    "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
+    "CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
+    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]",
+    "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]",
+    "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]"
+  }
+}
+```
+</Tip>
+
+### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse
+
+<Note>
+**Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it,
+multi-turn conversations re-prefill the whole history every turn.
+</Note>
+
+Claude Code prepends a per-request attribution block to the start of the system prompt, of the form
+`x-anthropic-billing-header: cc_version=<ver>.<per-request-hash>; cc_entrypoint=...; cch=<hash>;`. The
+per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse
+the short prefix before that hash and re-prefills the system prompt plus the entire conversation history
+on every turn.
+
+Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt.
+This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when
+routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)).
+
+<Note>
+`CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers
+autoupdater/telemetry/error reporting. The attribution header is a separate code path; use
+`CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it.
+</Note>
+
+## Troubleshooting
+
+**Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL`
+matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable
+and not behind a proxy that blocks the connection.
+
+**`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and
+serves whatever model was loaded at startup, so a 404 usually means the request did not reach the
+`/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and
+that the server finished loading.
+
+**Tool calls not working / returned as raw text** — Launch the server with the correct
+`--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted
+but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute
+them.
+
+**Slow / re-prefills the whole history every turn** — You are missing
+`CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt
+defeats radix prefix-cache reuse; see the section above.
+
+**Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context
+beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context
+is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend).
+
+## Parameters
+
+The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the
+[Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list.
+
+Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible
+endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models,
+`enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for
+the reasoning-parser / chat-template mapping.

docs_new/docs/basic_usage/aws_sagemaker.mdx

@@ -0,0 +1,142 @@
+---
+title: "Amazon SageMaker AI"
+description: "Deploy SGLang on Amazon SageMaker AI endpoints using the AWS Deep Learning Container."
+---
+
+Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoints using the
+[AWS Deep Learning Container (DLC)](https://aws.github.io/deep-learning-containers/sglang/) for SGLang.
+The SageMaker image variant accepts model configuration via environment variables and serves on port 8080.
+
+This guide uses the pre-built DLC image. To build and deploy your own container instead, see
+[Method 7: Run on AWS SageMaker](/docs/get-started/install#more-3) in the installation guide.
+
+## Container image
+
+AWS publishes pre-built, security-patched SGLang DLCs. The SageMaker GPU image is available from the
+Amazon ECR registry (account `763104351884`) in each supported region. For example, in `us-west-2`:
+
+```text
+763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0
+```
+
+For the full list of image tags, see the
+[Available DLC Images](https://aws.github.io/deep-learning-containers/reference/available_images/) reference,
+and for region-specific account IDs and supported regions, see
+[Region Availability](https://aws.github.io/deep-learning-containers/reference/region_availability/).
+
+## Specifying the model
+
+The SageMaker image resolves the model in this order:
+
+1. **`SM_SGLANG_MODEL_PATH` environment variable** — explicit Hugging Face ID or path.
+2. **`/opt/ml/model`** — when SageMaker mounts model artifacts via `ModelDataUrl` or `ModelDataSource`,
+   the entrypoint uses this path by default.
+
+For gated models, also pass `HF_TOKEN`.
+
+Any `SM_SGLANG_*` environment variable is converted to a `--<name>` SGLang server argument
+(for example, `SM_SGLANG_CONTEXT_LENGTH=4096` becomes `--context-length 4096`).
+
+## Deploy with the SageMaker Python SDK
+
+```python
+from sagemaker.model import Model
+from sagemaker.predictor import Predictor
+from sagemaker.serializers import JSONSerializer
+
+model = Model(
+    image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
+    role="arn:aws:iam::<account_id>:role/<role_name>",
+    predictor_cls=Predictor,
+    env={"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
+)
+
+predictor = model.deploy(
+    instance_type="ml.g5.2xlarge",
+    initial_instance_count=1,
+    inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1",
+    serializer=JSONSerializer(),
+)
+
+response = predictor.predict({
+    "model": "openai/gpt-oss-20b",
+    "messages": [{"role": "user", "content": "What is deep learning?"}],
+    "max_tokens": 256,
+})
+print(response)
+
+# Cleanup
+predictor.delete_model()
+predictor.delete_endpoint(delete_endpoint_config=True)
+```
+
+## Deploy with Boto3
+
+```python
+import json
+import boto3
+
+sm = boto3.client("sagemaker")
+smrt = boto3.client("sagemaker-runtime")
+
+sm.create_model(
+    ModelName="sglang-model",
+    PrimaryContainer={
+        "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
+        "Environment": {"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
+    },
+    ExecutionRoleArn="arn:aws:iam::<account_id>:role/<role_name>",
+)
+
+sm.create_endpoint_config(
+    EndpointConfigName="sglang-config",
+    ProductionVariants=[{
+        "VariantName": "default",
+        "ModelName": "sglang-model",
+        "InstanceType": "ml.g5.2xlarge",
+        "InitialInstanceCount": 1,
+        "InferenceAmiVersion": "al2023-ami-sagemaker-inference-gpu-4-1",
+    }],
+)
+
+sm.create_endpoint(EndpointName="sglang-endpoint", EndpointConfigName="sglang-config")
+sm.get_waiter("endpoint_in_service").wait(EndpointName="sglang-endpoint")
+
+resp = smrt.invoke_endpoint(
+    EndpointName="sglang-endpoint",
+    ContentType="application/json",
+    Body=json.dumps({
+        "model": "openai/gpt-oss-20b",
+        "messages": [{"role": "user", "content": "What is deep learning?"}],
+        "max_tokens": 256,
+    }),
+)
+print(json.loads(resp["Body"].read()))
+
+# Cleanup
+sm.delete_endpoint(EndpointName="sglang-endpoint")
+sm.delete_endpoint_config(EndpointConfigName="sglang-config")
+sm.delete_model(ModelName="sglang-model")
+```
+
+## Model artifacts
+
+When `ModelDataUrl` (or `ModelDataSource`) points to a tarball or S3 prefix, SageMaker mounts the contents
+at `/opt/ml/model`. The entrypoint defaults `--model-path` to that location, so `SM_SGLANG_MODEL_PATH`
+can be omitted:
+
+```text
+model.tar.gz
+├── config.json              # standard model files (Hugging Face layout)
+├── tokenizer.json
+└── *.safetensors
+```
+
+## Notes
+
+- GPU deployments require `inference_ami_version` — the default SageMaker host AMI has incompatible NVIDIA
+  drivers for CUDA 13 images. See the
+  [ProductionVariant API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html)
+  for valid values.
+- The endpoint exposes an OpenAI-compatible API, so the request body matches the SGLang server's
+  `/v1/chat/completions` schema.

docs_new/docs/basic_usage/native_api.ipynb

@@ -0,0 +1,675 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SGLang Native APIs\n",
+    "\n",
+    "Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:\n",
+    "\n",
+    "- `/generate` (text generation model)\n",
+    "- `/get_model_info`\n",
+    "- `/server_info`\n",
+    "- `/health`\n",
+    "- `/health_generate`\n",
+    "- `/flush_cache`\n",
+    "- `/update_weights`\n",
+    "- `/encode`(embedding model)\n",
+    "- `/v1/rerank`(cross encoder rerank model)\n",
+    "- `/v1/score`(decoder-only scoring)\n",
+    "- `/classify`(reward model)\n",
+    "- `/start_expert_distribution_record`\n",
+    "- `/stop_expert_distribution_record`\n",
+    "- `/dump_expert_distribution_record`\n",
+    "- `/tokenize`\n",
+    "- `/detokenize`\n",
+    "- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)\n",
+    "\n",
+    "We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generate (text generation model)\n",
+    "Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](sampling_params.md)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/generate\"\n",
+    "data = {\"text\": \"What is the capital of France?\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Get Model Info\n",
+    "\n",
+    "Get the information of the model.\n",
+    "\n",
+    "- `model_path`: The path/name of the model.\n",
+    "- `is_generation`: Whether the model is used as generation model or embedding model.\n",
+    "- `tokenizer_path`: The path/name of the tokenizer.\n",
+    "- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.\n",
+    "- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.\n",
+    "- `has_image_understanding`: Whether the model has image-understanding capability.\n",
+    "- `has_audio_understanding`: Whether the model has audio-understanding capability.\n",
+    "- `model_type`: The model type from the HuggingFace config (e.g., \"qwen2\", \"llama\").\n",
+    "- `architectures`: The model architectures from the HuggingFace config (e.g., [\"Qwen2ForCausalLM\"])."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/get_model_info\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "response_json = response.json()\n",
+    "print_highlight(response_json)\n",
+    "assert response_json[\"model_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "assert response_json[\"is_generation\"] is True\n",
+    "assert response_json[\"tokenizer_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "assert response_json[\"preferred_sampling_params\"] is None\n",
+    "assert response_json.keys() == {\n",
+    "    \"model_path\",\n",
+    "    \"is_generation\",\n",
+    "    \"tokenizer_path\",\n",
+    "    \"preferred_sampling_params\",\n",
+    "    \"weight_version\",\n",
+    "    \"has_image_understanding\",\n",
+    "    \"has_audio_understanding\",\n",
+    "    \"model_type\",\n",
+    "    \"architectures\",\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Get Server Info\n",
+    "Gets the server information including CLI arguments, token limits, and memory pool sizes.\n",
+    "- Note: `get_server_info` merges the following deprecated endpoints:\n",
+    "  - `get_server_args`\n",
+    "  - `get_memory_pool_size`\n",
+    "  - `get_max_total_num_tokens`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/server_info\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Health Check\n",
+    "- `/health`: Check the health of the server.\n",
+    "- `/health_generate`: Check the health of the server by generating one token."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/health_generate\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/health\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Flush Cache\n",
+    "\n",
+    "Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.\n",
+    "\n",
+    "Parameters:\n",
+    "- `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.\n",
+    "\n",
+    "```bash\n",
+    "# With timeout (wait up to 30s for idle state)\n",
+    "curl -s -X POST \"http://127.0.0.1:30000/flush_cache?timeout=30\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/flush_cache\"\n",
+    "\n",
+    "response = requests.post(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Update Weights From Disk\n",
+    "\n",
+    "Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.\n",
+    "\n",
+    "SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# successful update with same architecture and size\n",
+    "\n",
+    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
+    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.text)\n",
+    "assert response.json()[\"success\"] is True\n",
+    "assert response.json()[\"message\"] == \"Succeeded to update model weights.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# failed update with different parameter size or wrong name\n",
+    "\n",
+    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
+    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct-wrong\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "print_highlight(response_json)\n",
+    "assert response_json[\"success\"] is False\n",
+    "assert response_json[\"message\"] == (\n",
+    "    \"Failed to get weights iterator: \"\n",
+    "    \"qwen/qwen2.5-0.5b-instruct-wrong\"\n",
+    "    \" (repository not found).\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Encode (embedding model)\n",
+    "\n",
+    "Encode text into embeddings. Note that this API is only available for [embedding models](openai_api_embeddings.ipynb) and will raise an error for generation models.\n",
+    "Therefore, we launch a new server to server an embedding model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "embedding_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
+    "    --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# successful encode for embedding model\n",
+    "\n",
+    "url = f\"http://localhost:{port}/encode\"\n",
+    "data = {\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"text\": \"Once upon a time\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## v1/rerank (cross encoder rerank model)\n",
+    "Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reranker_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \\\n",
+    "    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=reranker_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# compute rerank scores for query and documents\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/rerank\"\n",
+    "data = {\n",
+    "    \"model\": \"BAAI/bge-reranker-v2-m3\",\n",
+    "    \"query\": \"what is panda?\",\n",
+    "    \"documents\": [\n",
+    "        \"hi\",\n",
+    "        \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\",\n",
+    "    ],\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "for item in response_json:\n",
+    "    print_highlight(f\"Score: {item['score']:.2f} - Document: '{item['document']}'\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(reranker_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## v1/score (decoder-only scoring)\n",
+    "\n",
+    "Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.\n",
+    "\n",
+    "Parameters:\n",
+    "- `query`: Query text\n",
+    "- `items`: Item text(s) to score\n",
+    "- `label_token_ids`: Token IDs to compute probabilities for\n",
+    "- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)\n",
+    "- `item_first`: Whether items come first in concatenation order (default: False)\n",
+    "- `model`: Model name\n",
+    "\n",
+    "The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "score_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    "    --host 0.0.0.0 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=score_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Score the probability of different completions given a query\n",
+    "query = \"The capital of France is\"\n",
+    "items = [\"Paris\", \"London\", \"Berlin\"]\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/score\"\n",
+    "data = {\n",
+    "    \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    \"query\": query,\n",
+    "    \"items\": items,\n",
+    "    \"label_token_ids\": [9454, 2753],  # e.g. \"Yes\" and \"No\" token ids\n",
+    "    \"apply_softmax\": True,  # Normalize probabilities to sum to 1\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "\n",
+    "# Display scores for each item\n",
+    "for item, scores in zip(items, response_json[\"scores\"]):\n",
+    "    print_highlight(f\"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(score_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Classify (reward model)\n",
+    "\n",
+    "SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Note that SGLang now treats embedding models and reward models as the same type of models.\n",
+    "# This will be updated in the future.\n",
+    "\n",
+    "reward_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=reward_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "PROMPT = (\n",
+    "    \"What is the range of the numeric output of a sigmoid node in a neural network?\"\n",
+    ")\n",
+    "\n",
+    "RESPONSE1 = \"The output of a sigmoid node is bounded between -1 and 1.\"\n",
+    "RESPONSE2 = \"The output of a sigmoid node is bounded between 0 and 1.\"\n",
+    "\n",
+    "CONVS = [\n",
+    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE1}],\n",
+    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE2}],\n",
+    "]\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\")\n",
+    "prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)\n",
+    "\n",
+    "url = f\"http://localhost:{port}/classify\"\n",
+    "data = {\"model\": \"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\", \"text\": prompts}\n",
+    "\n",
+    "responses = requests.post(url, json=data).json()\n",
+    "for response in responses:\n",
+    "    print_highlight(f\"reward: {response['embedding'][0]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(reward_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Capture expert selection distribution in MoE models\n",
+    "\n",
+    "SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.\n",
+    "\n",
+    "*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "expert_record_server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=expert_record_server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(f\"http://localhost:{port}/start_expert_distribution_record\")\n",
+    "print_highlight(response)\n",
+    "\n",
+    "url = f\"http://localhost:{port}/generate\"\n",
+    "data = {\"text\": \"What is the capital of France?\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())\n",
+    "\n",
+    "response = requests.post(f\"http://localhost:{port}/stop_expert_distribution_record\")\n",
+    "print_highlight(response)\n",
+    "\n",
+    "response = requests.post(f\"http://localhost:{port}/dump_expert_distribution_record\")\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(expert_record_server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tokenize/Detokenize Example (Round Trip)\n",
+    "\n",
+    "This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_free_server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=tokenizer_free_server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "from sglang.utils import print_highlight\n",
+    "\n",
+    "base_url = f\"http://localhost:{port}\"\n",
+    "tokenize_url = f\"{base_url}/tokenize\"\n",
+    "detokenize_url = f\"{base_url}/detokenize\"\n",
+    "\n",
+    "model_name = \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "input_text = \"SGLang provides efficient tokenization endpoints.\"\n",
+    "print_highlight(f\"Original Input Text:\\n'{input_text}'\")\n",
+    "\n",
+    "# --- tokenize the input text ---\n",
+    "tokenize_payload = {\n",
+    "    \"model\": model_name,\n",
+    "    \"prompt\": input_text,\n",
+    "    \"add_special_tokens\": False,\n",
+    "}\n",
+    "try:\n",
+    "    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)\n",
+    "    tokenize_response.raise_for_status()\n",
+    "    tokenization_result = tokenize_response.json()\n",
+    "    token_ids = tokenization_result.get(\"tokens\")\n",
+    "\n",
+    "    if not token_ids:\n",
+    "        raise ValueError(\"Tokenization returned empty tokens.\")\n",
+    "\n",
+    "    print_highlight(f\"\\nTokenized Output (IDs):\\n{token_ids}\")\n",
+    "    print_highlight(f\"Token Count: {tokenization_result.get('count')}\")\n",
+    "    print_highlight(f\"Max Model Length: {tokenization_result.get('max_model_len')}\")\n",
+    "\n",
+    "    # --- detokenize the obtained token IDs ---\n",
+    "    detokenize_payload = {\n",
+    "        \"model\": model_name,\n",
+    "        \"tokens\": token_ids,\n",
+    "        \"skip_special_tokens\": True,\n",
+    "    }\n",
+    "\n",
+    "    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)\n",
+    "    detokenize_response.raise_for_status()\n",
+    "    detokenization_result = detokenize_response.json()\n",
+    "    reconstructed_text = detokenization_result.get(\"text\")\n",
+    "\n",
+    "    print_highlight(f\"\\nDetokenized Output (Text):\\n'{reconstructed_text}'\")\n",
+    "\n",
+    "    if input_text == reconstructed_text:\n",
+    "        print_highlight(\n",
+    "            \"\\nRound Trip Successful: Original and reconstructed text match.\"\n",
+    "        )\n",
+    "    else:\n",
+    "        print_highlight(\n",
+    "            \"\\nRound Trip Mismatch: Original and reconstructed text differ.\"\n",
+    "        )\n",
+    "\n",
+    "except requests.exceptions.RequestException as e:\n",
+    "    print_highlight(f\"\\nHTTP Request Error: {e}\")\n",
+    "except Exception as e:\n",
+    "    print_highlight(f\"\\nAn error occurred: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(tokenizer_free_server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs_new/docs/basic_usage/native_api.mdx

@@ -0,0 +1,448 @@
+---
+title: "SGLang Native APIs"
+metatags:
+    description: "SGLang native server APIs for text generation, embedding, reranking, model info, cache management, and more."
+---
+Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:
+
+- `/generate` (text generation model)
+- `/get_model_info`
+- `/server_info`
+- `/health`
+- `/health_generate`
+- `/flush_cache`
+- `/update_weights`
+- `/encode`(embedding model)
+- `/v1/rerank`(cross encoder rerank model)
+- `/v1/score`(decoder-only scoring)
+- `/classify`(reward model)
+- `/start_expert_distribution_record`
+- `/stop_expert_distribution_record`
+- `/dump_expert_distribution_record`
+- `/tokenize`
+- `/detokenize`
+- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)
+
+We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.
+
+## Launch A Server
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=server_process)
+```
+
+## Generate (text generation model)
+Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params).
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/generate"
+data = {"text": "What is the capital of France?"}
+
+response = requests.post(url, json=data)
+print_highlight(response.json())
+```
+
+## Get Model Info
+
+Get the information of the model.
+
+- `model_path`: The path/name of the model.
+- `is_generation`: Whether the model is used as generation model or embedding model.
+- `tokenizer_path`: The path/name of the tokenizer.
+- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.
+- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.
+- `has_image_understanding`: Whether the model has image-understanding capability.
+- `has_audio_understanding`: Whether the model has audio-understanding capability.
+- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
+- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
+
+```python Example
+url = f"http://localhost:{port}/get_model_info"
+
+response = requests.get(url)
+response_json = response.json()
+print_highlight(response_json)
+assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
+assert response_json["is_generation"] is True
+assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
+assert response_json["preferred_sampling_params"] is None
+assert response_json.keys() == {
+    "model_path",
+    "is_generation",
+    "tokenizer_path",
+    "preferred_sampling_params",
+    "weight_version",
+    "has_image_understanding",
+    "has_audio_understanding",
+    "model_type",
+    "architectures",
+}
+```
+
+## Get Server Info
+Gets the server information including CLI arguments, token limits, and memory pool sizes.
+- Note: `get_server_info` merges the following deprecated endpoints:
+  - `get_server_args`
+  - `get_memory_pool_size`
+  - `get_max_total_num_tokens`
+
+```python Example
+url = f"http://localhost:{port}/server_info"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+## Health Check
+- `/health`: Check the health of the server.
+- `/health_generate`: Check the health of the server by generating one token.
+
+```python Example
+url = f"http://localhost:{port}/health_generate"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+```python Example
+url = f"http://localhost:{port}/health"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+## Flush Cache
+
+Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.
+
+Parameters:
+- `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.
+
+```bash Command
+# With timeout (wait up to 30s for idle state)
+curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30"
+```
+
+```python Example
+url = f"http://localhost:{port}/flush_cache"
+
+response = requests.post(url)
+print_highlight(response.text)
+```
+
+## Update Weights From Disk
+
+Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.
+
+SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).
+
+```python Example
+# successful update with same architecture and size
+
+url = f"http://localhost:{port}/update_weights_from_disk"
+data = {"model_path": "qwen/qwen2.5-0.5b-instruct"}
+
+response = requests.post(url, json=data)
+print_highlight(response.text)
+assert response.json()["success"] is True
+assert response.json()["message"] == "Succeeded to update model weights."
+```
+
+```python Example
+# failed update with different parameter size or wrong name
+
+url = f"http://localhost:{port}/update_weights_from_disk"
+data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+print_highlight(response_json)
+assert response_json["success"] is False
+assert response_json["message"] == (
+    "Failed to get weights iterator: "
+    "qwen/qwen2.5-0.5b-instruct-wrong"
+    " (repository not found)."
+)
+```
+
+```python Example
+terminate_process(server_process)
+```
+
+## Encode (embedding model)
+
+Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models.
+Therefore, we launch a new server to server an embedding model.
+
+```python Example
+embedding_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
+    --host 0.0.0.0 --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=embedding_process)
+```
+
+```python Example
+# successful encode for embedding model
+
+url = f"http://localhost:{port}/encode"
+data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")
+```
+
+```python Example
+terminate_process(embedding_process)
+```
+
+## v1/rerank (cross encoder rerank model)
+Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.
+
+```python Example
+reranker_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
+    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=reranker_process)
+```
+
+```python Example
+# compute rerank scores for query and documents
+
+url = f"http://localhost:{port}/v1/rerank"
+data = {
+    "model": "BAAI/bge-reranker-v2-m3",
+    "query": "what is panda?",
+    "documents": [
+        "hi",
+        "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.",
+    ],
+}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+for item in response_json:
+    print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
+```
+
+```python Example
+terminate_process(reranker_process)
+```
+
+## v1/score (decoder-only scoring)
+
+Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.
+
+Parameters:
+- `query`: Query text
+- `items`: Item text(s) to score
+- `label_token_ids`: Token IDs to compute probabilities for
+- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)
+- `item_first`: Whether items come first in concatenation order (default: False)
+- `model`: Model name
+
+The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`.
+
+```python Example
+score_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
+    --host 0.0.0.0 --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=score_process)
+```
+
+```python Example
+# Score the probability of different completions given a query
+query = "The capital of France is"
+items = ["Paris", "London", "Berlin"]
+
+url = f"http://localhost:{port}/v1/score"
+data = {
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "query": query,
+    "items": items,
+    "label_token_ids": [9454, 2753],  # e.g. "Yes" and "No" token ids
+    "apply_softmax": True,  # Normalize probabilities to sum to 1
+}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+
+# Display scores for each item
+for item, scores in zip(items, response_json["scores"]):
+    print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}")
+```
+
+```python Example
+terminate_process(score_process)
+```
+
+## Classify (reward model)
+
+SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.
+
+```python Example
+# Note that SGLang now treats embedding models and reward models as the same type of models.
+# This will be updated in the future.
+
+reward_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=reward_process)
+```
+
+```python Example
+from transformers import AutoTokenizer
+
+PROMPT = (
+    "What is the range of the numeric output of a sigmoid node in a neural network?"
+)
+
+RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
+RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."
+
+CONVS = [
+    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
+    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
+]
+
+tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
+prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)
+
+url = f"http://localhost:{port}/classify"
+data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}
+
+responses = requests.post(url, json=data).json()
+for response in responses:
+    print_highlight(f"reward: {response['embedding'][0]}")
+```
+
+```python Example
+terminate_process(reward_process)
+```
+
+## Capture expert selection distribution in MoE models
+
+SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.
+
+*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*
+
+```python Example
+expert_record_server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=expert_record_server_process)
+```
+
+```python Example
+response = requests.post(f"http://localhost:{port}/start_expert_distribution_record")
+print_highlight(response)
+
+url = f"http://localhost:{port}/generate"
+data = {"text": "What is the capital of France?"}
+
+response = requests.post(url, json=data)
+print_highlight(response.json())
+
+response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record")
+print_highlight(response)
+
+response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record")
+print_highlight(response)
+```
+
+```python Example
+terminate_process(expert_record_server_process)
+```
+
+## Tokenize/Detokenize Example (Round Trip)
+
+This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.
+
+```python Example
+tokenizer_free_server_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
+""")
+
+wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process)
+```
+
+```python Example
+import requests
+from sglang.utils import print_highlight
+
+base_url = f"http://localhost:{port}"
+tokenize_url = f"{base_url}/tokenize"
+detokenize_url = f"{base_url}/detokenize"
+
+model_name = "qwen/qwen2.5-0.5b-instruct"
+input_text = "SGLang provides efficient tokenization endpoints."
+print_highlight(f"Original Input Text:\n'{input_text}'")
+
+# --- tokenize the input text ---
+tokenize_payload = {
+    "model": model_name,
+    "prompt": input_text,
+    "add_special_tokens": False,
+}
+try:
+    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)
+    tokenize_response.raise_for_status()
+    tokenization_result = tokenize_response.json()
+    token_ids = tokenization_result.get("tokens")
+
+    if not token_ids:
+        raise ValueError("Tokenization returned empty tokens.")
+
+    print_highlight(f"\nTokenized Output (IDs):\n{token_ids}")
+    print_highlight(f"Token Count: {tokenization_result.get('count')}")
+    print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}")
+
+    # --- detokenize the obtained token IDs ---
+    detokenize_payload = {
+        "model": model_name,
+        "tokens": token_ids,
+        "skip_special_tokens": True,
+    }
+
+    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)
+    detokenize_response.raise_for_status()
+    detokenization_result = detokenize_response.json()
+    reconstructed_text = detokenization_result.get("text")
+
+    print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'")
+
+    if input_text == reconstructed_text:
+        print_highlight(
+            "\nRound Trip Successful: Original and reconstructed text match."
+        )
+    else:
+        print_highlight(
+            "\nRound Trip Mismatch: Original and reconstructed text differ."
+        )
+
+except requests.exceptions.RequestException as e:
+    print_highlight(f"\nHTTP Request Error: {e}")
+except Exception as e:
+    print_highlight(f"\nAn error occurred: {e}")
+```
+
+```python Example
+terminate_process(tokenizer_free_server_process)
+```

docs_new/docs/basic_usage/offline_engine_api.ipynb

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Offline Engine API\n",
+    "\n",
+    "SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:\n",
+    "\n",
+    "- Offline Batch Inference\n",
+    "- Custom Server on Top of the Engine\n",
+    "\n",
+    "This document focuses on the offline batch inference, demonstrating four different inference modes:\n",
+    "\n",
+    "- Non-streaming synchronous generation\n",
+    "- Streaming synchronous generation\n",
+    "- Non-streaming asynchronous generation\n",
+    "- Streaming asynchronous generation\n",
+    "\n",
+    "Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Nest Asyncio\n",
+    "Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:\n",
+    "```python\n",
+    "import nest_asyncio\n",
+    "\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Advanced Usage\n",
+    "\n",
+    "The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/blob/main/examples/runtime/hidden_states). \n",
+    "\n",
+    "Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Ray Integration\n",
+    "\n",
+    "When running in a Ray cluster, you can use `RayEngine` with a custom placement group for fine-grained GPU placement control.\n",
+    "\n",
+    "### Custom Placement Groups\n",
+    "\n",
+    "Pass a `placement_group` with 1-GPU-per-bundle bundles to control exactly which GPUs are used. Each bundle should have exactly 1 GPU for deterministic mapping.\n",
+    "\n",
+    "```python\n",
+    "import ray\n",
+    "from ray.util.placement_group import placement_group\n",
+    "from sglang.srt.ray.engine import RayEngine\n",
+    "\n",
+    "ray.init()\n",
+    "\n",
+    "# Create placement group with specific GPU bundles\n",
+    "pg = placement_group(\n",
+    "    [{\"GPU\": 1} for _ in range(4)],  # 4 bundles, each with 1 GPU\n",
+    "    strategy=\"STRICT_PACK\",\n",
+    ")\n",
+    "ray.get(pg.ready())\n",
+    "\n",
+    "# Launch RayEngine on custom placement group\n",
+    "engine = RayEngine(\n",
+    "    model_path=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
+    "    tp_size=4,\n",
+    "    use_ray=True,\n",
+    "    placement_group=pg,\n",
+    ")\n",
+    "\n",
+    "# Optional: specify exact bundle indices via environment variable\n",
+    "# export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,3\"\n",
+    "```\n",
+    "\n",
+    "### Bundle Index Control\n",
+    "\n",
+    "Use `SGLANG_RAY_BUNDLE_INDICES` environment variable to specify which placement group bundles to use for each worker rank. This enables:\n",
+    "- Skipping unhealthy GPUs\n",
+    "- Topology-aware placement (e.g., NVLink-connected GPUs)\n",
+    "- Non-sequential bundle assignment\n",
+    "\n",
+    "```bash\n",
+    "# Use bundles 0,1,2,7 (skip bundles 3-6) for tp_size=4\n",
+    "export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,7\"\n",
+    "\n",
+    "# Place workers on NVLink-connected GPUs\n",
+    "export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,3\"\n",
+    "```\n",
+    "\n",
+    "The number of indices must match `world_size` (`tp_size * pp_size * dp_size`, or `tp_size * pp_size` when `enable_dp_attention=True`)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Offline Batch Inference\n",
+    "\n",
+    "SGLang offline engine supports batch inference with efficient scheduling."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# launch the offline engine\n",
+    "import asyncio\n",
+    "\n",
+    "import sglang as sgl\n",
+    "import sglang.test.doc_patch  # noqa: F401\n",
+    "from sglang.utils import async_stream_and_merge, stream_and_merge\n",
+    "\n",
+    "llm = sgl.Engine(model_path=\"qwen/qwen2.5-0.5b-instruct\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Non-streaming Synchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Hello, my name is\",\n",
+    "    \"The president of the United States is\",\n",
+    "    \"The capital of France is\",\n",
+    "    \"The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming Synchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.2,\n",
+    "    \"top_p\": 0.9,\n",
+    "}\n",
+    "\n",
+    "print(\"\\n=== Testing synchronous streaming generation with overlap removal ===\\n\")\n",
+    "\n",
+    "for prompt in prompts:\n",
+    "    print(f\"Prompt: {prompt}\")\n",
+    "    merged_output = stream_and_merge(llm, prompt, sampling_params)\n",
+    "    print(\"Generated text:\", merged_output)\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Non-streaming Asynchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "print(\"\\n=== Testing asynchronous batch generation ===\")\n",
+    "\n",
+    "\n",
+    "async def main():\n",
+    "    outputs = await llm.async_generate(prompts, sampling_params)\n",
+    "\n",
+    "    for prompt, output in zip(prompts, outputs):\n",
+    "        print(f\"\\nPrompt: {prompt}\")\n",
+    "        print(f\"Generated text: {output['text']}\")\n",
+    "\n",
+    "\n",
+    "asyncio.run(main())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming Asynchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "print(\"\\n=== Testing asynchronous streaming generation (no repeats) ===\")\n",
+    "\n",
+    "\n",
+    "async def main():\n",
+    "    for prompt in prompts:\n",
+    "        print(f\"\\nPrompt: {prompt}\")\n",
+    "        print(\"Generated text: \", end=\"\", flush=True)\n",
+    "\n",
+    "        # Replace direct calls to async_generate with our custom overlap-aware version\n",
+    "        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):\n",
+    "            print(cleaned_chunk, end=\"\", flush=True)\n",
+    "\n",
+    "        print()  # New line after each prompt\n",
+    "\n",
+    "\n",
+    "asyncio.run(main())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/offline_engine_api.mdx

@@ -0,0 +1,143 @@
+---
+title: "Offline Engine API"
+metatags:
+    description: "Use SGLang's offline engine for direct batch inference without HTTP server overhead. Supports sync/async and streaming modes."
+---
+SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:
+
+- Offline Batch Inference
+- Custom Server on Top of the Engine
+
+This document focuses on the offline batch inference, demonstrating four different inference modes:
+
+- Non-streaming synchronous generation
+- Streaming synchronous generation
+- Non-streaming asynchronous generation
+- Streaming asynchronous generation
+
+Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).
+
+## Nest Asyncio
+Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:
+```python Example
+import nest_asyncio
+
+nest_asyncio.apply()
+
+```
+
+## Advanced Usage
+
+The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states).
+
+Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases.
+
+## Offline Batch Inference
+
+SGLang offline engine supports batch inference with efficient scheduling.
+
+```python Example
+# launch the offline engine
+import asyncio
+
+import sglang as sgl
+import sglang.test.doc_patch
+from sglang.utils import async_stream_and_merge, stream_and_merge
+
+llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
+```
+
+### Non-streaming Synchronous Generation
+
+```python Example
+prompts = [
+    "Hello, my name is",
+    "The president of the United States is",
+    "The capital of France is",
+    "The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### Streaming Synchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {
+    "temperature": 0.2,
+    "top_p": 0.9,
+}
+
+print("\n=== Testing synchronous streaming generation with overlap removal ===\n")
+
+for prompt in prompts:
+    print(f"Prompt: {prompt}")
+    merged_output = stream_and_merge(llm, prompt, sampling_params)
+    print("Generated text:", merged_output)
+    print()
+```
+
+### Non-streaming Asynchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+print("\n=== Testing asynchronous batch generation ===")
+
+async def main():
+    outputs = await llm.async_generate(prompts, sampling_params)
+
+    for prompt, output in zip(prompts, outputs):
+        print(f"\nPrompt: {prompt}")
+        print(f"Generated text: {output['text']}")
+
+asyncio.run(main())
+```
+
+### Streaming Asynchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+print("\n=== Testing asynchronous streaming generation (no repeats) ===")
+
+async def main():
+    for prompt in prompts:
+        print(f"\nPrompt: {prompt}")
+        print("Generated text: ", end="", flush=True)
+
+        # Replace direct calls to async_generate with our custom overlap-aware version
+        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):
+            print(cleaned_chunk, end="", flush=True)
+
+        print()  # New line after each prompt
+
+asyncio.run(main())
+```
+
+```python Example
+llm.shutdown()
+```

docs_new/docs/basic_usage/ollama_api.mdx

@@ -0,0 +1,157 @@
+---
+title: "Ollama-Compatible API"
+metatags:
+    description: "SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend."
+---
+SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend.
+
+## Prerequisites
+
+<CodeGroup>
+```bash Command
+# Install the Ollama Python library (for Python client usage)
+pip install ollama
+```
+</CodeGroup>
+
+<Note>You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client.</Note>
+
+## Endpoints
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Endpoint</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Method</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET, HEAD</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Health check for Ollama CLI</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/tags`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List available models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/chat`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chat completions (streaming & non-streaming)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/generate`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Text generation (streaming & non-streaming)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/show`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model information</td>
+    </tr>
+  </tbody>
+</table>
+
+## Quick Start
+
+### 1. Launch SGLang Server
+
+<CodeGroup>
+```bash Command
+python -m sglang.launch_server \
+    --model Qwen/Qwen2.5-1.5B-Instruct \
+    --port 30001 \
+    --host 0.0.0.0
+```
+</CodeGroup>
+
+<Note>The model name used with `ollama run` must match exactly what you passed to `--model`.</Note>
+
+### 2. Use Ollama CLI
+
+<CodeGroup>
+```bash Command
+# List available models
+OLLAMA_HOST=http://localhost:30001 ollama list
+
+# Interactive chat
+OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct"
+```
+</CodeGroup>
+
+If connecting to a remote server behind a firewall:
+
+<CodeGroup>
+```bash Command
+# SSH tunnel
+ssh -L 30001:localhost:30001 user@gpu-server -N &
+
+# Then use Ollama CLI as above
+OLLAMA_HOST=http://localhost:30001 ollama list
+```
+</CodeGroup>
+
+### 3. Use Ollama Python Library
+
+```python Example
+import ollama
+
+client = ollama.Client(host='http://localhost:30001')
+
+# Non-streaming
+response = client.chat(
+    model='Qwen/Qwen2.5-1.5B-Instruct',
+    messages=[{'role': 'user', 'content': 'Hello!'}]
+)
+print(response['message']['content'])
+
+# Streaming
+stream = client.chat(
+    model='Qwen/Qwen2.5-1.5B-Instruct',
+    messages=[{'role': 'user', 'content': 'Tell me a story'}],
+    stream=True
+)
+for chunk in stream:
+    print(chunk['message']['content'], end='', flush=True)
+```
+
+## Smart Router
+
+For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README).
+
+## Summary
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ollama API**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Familiar CLI/API that developers already know</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**SGLang Backend**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>High-performance inference engine</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Smart Router**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Intelligent routing - fast local for simple tasks, powerful remote for complex tasks</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/basic_usage/openai_api.mdx

@@ -0,0 +1,9 @@
+---
+title: "OpenAI-Compatible APIs"
+description: "Documentation for OpenAI-Compatible APIs"
+---
+- [Openai Api Completions](./openai_api_completions)
+- [Openai Api Vision](./openai_api_vision)
+- [Openai Api Embeddings](./openai_api_embeddings)
+
+For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api).

docs_new/docs/basic_usage/openai_api.rst

@@ -0,0 +1,9 @@
+OpenAI-Compatible APIs
+======================
+
+.. toctree::
+   :maxdepth: 1
+
+   openai_api_completions.ipynb
+   openai_api_vision.ipynb
+   openai_api_embeddings.ipynb

docs_new/docs/basic_usage/openai_api_completions.ipynb

@@ -0,0 +1,552 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Completions\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).\n",
+    "\n",
+    "This tutorial covers the following popular APIs:\n",
+    "\n",
+    "- `chat/completions`\n",
+    "- `completions`\n",
+    "\n",
+    "Check out other tutorials to learn about [vision APIs](openai_api_vision.ipynb) for vision-language models and [embedding APIs](openai_api_embeddings.ipynb) for embedding models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "print(f\"Server started on http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Chat Completions\n",
+    "\n",
+    "### Usage\n",
+    "\n",
+    "The server fully implements the OpenAI API.\n",
+    "It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.\n",
+    "You can also specify a custom chat template with `--chat-template` when launching the server."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Model Thinking/Reasoning Support\n",
+    "\n",
+    "Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.\n",
+    "\n",
+    "#### Supported Models and Configuration\n",
+    "\n",
+    "| Model Family | Chat Template Parameter | Reasoning Parser | Notes |\n",
+    "|--------------|------------------------|------------------|--------|\n",
+    "| DeepSeek-R1 (R1, R1-0528, R1-Distill) | `enable_thinking` | `--reasoning-parser deepseek-r1` | Standard reasoning models |\n",
+    "| DeepSeek-V3.1 | `thinking` | `--reasoning-parser deepseek-v3` | Hybrid model (thinking/non-thinking modes) |\n",
+    "| Qwen3 (standard) | `enable_thinking` | `--reasoning-parser qwen3` | Hybrid model (thinking/non-thinking modes) |\n",
+    "| Qwen3-Thinking | N/A (always enabled) | `--reasoning-parser qwen3-thinking` | Always generates reasoning |\n",
+    "| Kimi | N/A (always enabled) | `--reasoning-parser kimi` | Kimi thinking models |\n",
+    "| Gpt-Oss | N/A (always enabled) | `--reasoning-parser gpt-oss` | Gpt-Oss thinking models |\n",
+    "\n",
+    "#### Basic Usage\n",
+    "\n",
+    "To enable reasoning output, you need to:\n",
+    "1. Launch the server with the appropriate reasoning parser\n",
+    "2. Set the model-specific parameter in `chat_template_kwargs`\n",
+    "3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)\n",
+    "\n",
+    "**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Example: Qwen3 Models\n",
+    "\n",
+    "```python\n",
+    "# Launch server:\n",
+    "# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=\"EMPTY\",\n",
+    "    base_url=f\"http://127.0.0.1:30000/v1\",\n",
+    ")\n",
+    "\n",
+    "model = \"Qwen/Qwen3-4B\"\n",
+    "messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=model,\n",
+    "    messages=messages,\n",
+    "    extra_body={\n",
+    "        \"chat_template_kwargs\": {\"enable_thinking\": True},\n",
+    "        \"separate_reasoning\": True\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
+    "print(\"-\"*100)\n",
+    "print(\"Answer:\", response.choices[0].message.content)\n",
+    "```\n",
+    "\n",
+    "**ExampleOutput:**\n",
+    "```\n",
+    "Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.\n",
+    "\n",
+    "Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y. \n",
+    "...\n",
+    "Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.\n",
+    "\n",
+    "----------------------------------------------------------------------------------------------------\n",
+    "Answer: The word \"strawberry\" contains **three** letters 'r'. Here's the breakdown:\n",
+    "\n",
+    "1. **S-T-R-A-W-B-E-R-R-Y**  \n",
+    "   - The **third letter** is 'R'.  \n",
+    "   - The **eighth and ninth letters** are also 'R's.  \n",
+    "\n",
+    "Thus, the total count is **3**.  \n",
+    "\n",
+    "**Answer:** 3.\n",
+    "```\n",
+    "\n",
+    "**Note:** Setting `\"enable_thinking\": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Logit Bias Support\n",
+    "\n",
+    "SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:\n",
+    "\n",
+    "- **Positive values** (0 to 100) increase the likelihood of the token being selected\n",
+    "- **Negative values** (-100 to 0) decrease the likelihood of the token being selected\n",
+    "- **-100** effectively prevents the token from being generated\n",
+    "\n",
+    "The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Getting Token IDs\n",
+    "\n",
+    "To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:\n",
+    "\n",
+    "```python\n",
+    "# Get tokenizer to find token IDs\n",
+    "import tiktoken\n",
+    "\n",
+    "# For OpenAI models, use the appropriate encoding\n",
+    "tokenizer = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")  # or your model\n",
+    "\n",
+    "# Get token IDs for specific words\n",
+    "word = \"sunny\"\n",
+    "token_ids = tokenizer.encode(word)\n",
+    "print(f\"Token IDs for '{word}': {token_ids}\")\n",
+    "\n",
+    "# For SGLang models, you can access the tokenizer through the client\n",
+    "# and get token IDs for bias\n",
+    "```\n",
+    "\n",
+    "**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Example: DeepSeek-V3 Models\n",
+    "\n",
+    "DeepSeek-V3 models support thinking mode through the `thinking` parameter:\n",
+    "\n",
+    "```python\n",
+    "# Launch server:\n",
+    "# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8  --reasoning-parser deepseek-v3\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=\"EMPTY\",\n",
+    "    base_url=f\"http://127.0.0.1:30000/v1\",\n",
+    ")\n",
+    "\n",
+    "model = \"deepseek-ai/DeepSeek-V3.1\"\n",
+    "messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=model,\n",
+    "    messages=messages,\n",
+    "    extra_body={\n",
+    "        \"chat_template_kwargs\": {\"thinking\": True},\n",
+    "        \"separate_reasoning\": True\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
+    "print(\"-\"*100)\n",
+    "print(\"Answer:\", response.choices[0].message.content)\n",
+    "```\n",
+    "\n",
+    "**Example Output:**\n",
+    "```\n",
+    "Reasoning: First, the question is: \"How many r's are in 'strawberry'?\"\n",
+    "\n",
+    "I need to count the number of times the letter 'r' appears in the word \"strawberry\".\n",
+    "\n",
+    "Let me write out the word: S-T-R-A-W-B-E-R-R-Y.\n",
+    "\n",
+    "Now, I'll go through each letter and count the 'r's.\n",
+    "...\n",
+    "So, I have three 'r's in \"strawberry\".\n",
+    "\n",
+    "I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.\n",
+    "\n",
+    "Therefore, the answer should be 3.\n",
+    "----------------------------------------------------------------------------------------------------\n",
+    "Answer: The word \"strawberry\" contains **3** instances of the letter \"r\". Here's a breakdown for clarity:\n",
+    "\n",
+    "- The word is spelled: S-T-R-A-W-B-E-R-R-Y\n",
+    "- The \"r\" appears at the 3rd, 8th, and 9th positions.\n",
+    "```\n",
+    "\n",
+    "**Note:** DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example with logit_bias parameter\n",
+    "# Note: You need to get the actual token IDs from your tokenizer\n",
+    "# For demonstration, we'll use some example token IDs\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"Complete this sentence: The weather today is\"}\n",
+    "    ],\n",
+    "    temperature=0.7,\n",
+    "    max_tokens=20,\n",
+    "    logit_bias={\n",
+    "        \"12345\": 50,  # Increase likelihood of token ID 12345\n",
+    "        \"67890\": -50,  # Decrease likelihood of token ID 67890\n",
+    "        \"11111\": 25,  # Slightly increase likelihood of token ID 11111\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response with logit bias: {response.choices[0].message.content}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Parameters\n",
+    "\n",
+    "The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.\n",
+    "\n",
+    "SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": \"You are a knowledgeable historian who provides concise responses.\",\n",
+    "        },\n",
+    "        {\"role\": \"user\", \"content\": \"Tell me about ancient Rome\"},\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Ancient Rome was a civilization centered in Italy.\",\n",
+    "        },\n",
+    "        {\"role\": \"user\", \"content\": \"What were their major achievements?\"},\n",
+    "    ],\n",
+    "    temperature=0.3,  # Lower temperature for more focused responses\n",
+    "    max_tokens=128,  # Reasonable length for a concise response\n",
+    "    top_p=0.95,  # Slightly higher for better fluency\n",
+    "    presence_penalty=0.2,  # Mild penalty to avoid repetition\n",
+    "    frequency_penalty=0.2,  # Mild penalty for more natural language\n",
+    "    n=1,  # Single response is usually more stable\n",
+    "    seed=42,  # Keep for reproducibility\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Streaming mode is also supported."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Logit Bias Support\n",
+    "\n",
+    "The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "stream = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Say this is a test\"}],\n",
+    "    stream=True,\n",
+    ")\n",
+    "for chunk in stream:\n",
+    "    if chunk.choices[0].delta.content is not None:\n",
+    "        print(chunk.choices[0].delta.content, end=\"\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Returning Routed Experts (MoE Models)\n",
+    "\n",
+    "For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example with logit_bias parameter for completions API\n",
+    "# Note: You need to get the actual token IDs from your tokenizer\n",
+    "# For demonstration, we'll use some example token IDs\n",
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"The best programming language for AI is\",\n",
+    "    temperature=0.7,\n",
+    "    max_tokens=20,\n",
+    "    logit_bias={\n",
+    "        \"12345\": 75,  # Strongly favor token ID 12345\n",
+    "        \"67890\": -100,  # Completely avoid token ID 67890\n",
+    "        \"11111\": -25,  # Slightly discourage token ID 11111\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response with logit bias: {response.choices[0].text}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Completions\n",
+    "\n",
+    "### Usage\n",
+    "Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"List 3 countries and their capitals.\",\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    "    n=1,\n",
+    "    stop=None,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Parameters\n",
+    "\n",
+    "The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.\n",
+    "\n",
+    "Here is an example of a detailed completions request:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"Write a short story about a space explorer.\",\n",
+    "    temperature=0.7,  # Moderate temperature for creative writing\n",
+    "    max_tokens=150,  # Longer response for a story\n",
+    "    top_p=0.9,  # Balanced diversity in word choice\n",
+    "    stop=[\"\\n\\n\", \"THE END\"],  # Multiple stop sequences\n",
+    "    presence_penalty=0.3,  # Encourage novel elements\n",
+    "    frequency_penalty=0.3,  # Reduce repetitive phrases\n",
+    "    n=1,  # Generate one completion\n",
+    "    seed=123,  # For reproducible results\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Returning Routed Experts (MoE Models)\n",
+    "\n",
+    "For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Structured Outputs (JSON, Regex, EBNF)\n",
+    "\n",
+    "For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs.ipynb) for more details.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using LoRA Adapters\n",
+    "\n",
+    "SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.\n",
+    "\n",
+    "**Server Setup:**\n",
+    "```bash\n",
+    "python -m sglang.launch_server \\\n",
+    "    --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b\n",
+    "```\n",
+    "\n",
+    "For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora.ipynb).\n",
+    "\n",
+    "**API Call:**\n",
+    "\n",
+    "(Recommended) Use the `model:adapter` syntax to specify which adapter to use:\n",
+    "```python\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct:adapter_a\",  # ← base-model:adapter-name\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
+    "    max_tokens=50,\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "**Backward Compatible: Using `extra_body`**\n",
+    "\n",
+    "The old `extra_body` method is still supported for backward compatibility:\n",
+    "```python\n",
+    "# Backward compatible method\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
+    "    extra_body={\"lora_path\": \"adapter_a\"},  # ← old method\n",
+    "    max_tokens=50,\n",
+    ")\n",
+    "```\n",
+    "**Note:** When both `model:adapter` and `extra_body[\"lora_path\"]` are specified, the `model:adapter` syntax takes precedence."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_completions.mdx

@@ -0,0 +1,456 @@
+---
+title: "OpenAI APIs - Completions"
+metatags:
+    description: "This tutorial covers the following popular APIs: 'chat/completions' and 'completions'"
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).
+
+This tutorial covers the following popular APIs:
+
+- `chat/completions`
+- `completions`
+
+Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models.
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize.
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}")
+print(f"Server started on http://localhost:{port}")
+```
+
+## Chat Completions
+
+### Usage
+
+The server fully implements the OpenAI API.
+It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.
+You can also specify a custom chat template with `--chat-template` when launching the server.
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+)
+
+print_highlight(f"Response: {response}")
+```
+
+### Model Thinking/Reasoning Support
+
+Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.
+
+#### Supported Models and Configuration
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Chat Template Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Reasoning Parser</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1 (R1, R1-0528, R1-Distill)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-r1`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Standard reasoning models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-v3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3 (standard)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Thinking</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3-thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates reasoning</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser kimi`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi thinking models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Gpt-Oss</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser gpt-oss`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gpt-Oss thinking models</td>
+    </tr>
+  </tbody>
+</table>
+
+#### Basic Usage
+
+To enable reasoning output, you need to:
+1. Launch the server with the appropriate reasoning parser
+2. Set the model-specific parameter in `chat_template_kwargs`
+3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)
+
+<Note>
+**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.
+</Note>
+
+#### Example: Qwen3 Models
+
+```python Example
+# Launch server:
+# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3
+
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url=f"http://127.0.0.1:30000/v1",
+)
+
+model = "Qwen/Qwen3-4B"
+messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    extra_body={
+        "chat_template_kwargs": {"enable_thinking": True},
+        "separate_reasoning": True
+    }
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("-"*100)
+print("Answer:", response.choices[0].message.content)
+```
+
+**ExampleOutput:**
+```text Output
+Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.
+
+Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y.
+...
+Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.
+
+----------------------------------------------------------------------------------------------------
+Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown:
+
+1. **S-T-R-A-W-B-E-R-R-Y**
+   - The **third letter** is 'R'.
+   - The **eighth and ninth letters** are also 'R's.
+
+Thus, the total count is **3**.
+
+**Answer:** 3.
+```
+<Note>
+Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.
+</Note>
+
+#### Logit Bias Support
+
+SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:
+
+- **Positive values** (0 to 100) increase the likelihood of the token being selected
+- **Negative values** (-100 to 0) decrease the likelihood of the token being selected
+- **-100** effectively prevents the token from being generated
+
+The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).
+
+#### Getting Token IDs
+
+To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:
+
+```python Example
+# Get tokenizer to find token IDs
+import tiktoken
+
+# For OpenAI models, use the appropriate encoding
+tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")  # or your model
+
+# Get token IDs for specific words
+word = "sunny"
+token_ids = tokenizer.encode(word)
+print(f"Token IDs for '{word}': {token_ids}")
+
+# For SGLang models, you can access the tokenizer through the client
+# and get token IDs for bias
+```
+<Tip>
+**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.
+</Tip>
+
+#### Example: DeepSeek-V3 Models
+
+DeepSeek-V3 models support thinking mode through the `thinking` parameter:
+
+```python Example
+# Launch server:
+# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8  --reasoning-parser deepseek-v3
+
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url=f"http://127.0.0.1:30000/v1",
+)
+
+model = "deepseek-ai/DeepSeek-V3.1"
+messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    extra_body={
+        "chat_template_kwargs": {"thinking": True},
+        "separate_reasoning": True
+    }
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("-"*100)
+print("Answer:", response.choices[0].message.content)
+```
+
+**Example Output:**
+```text Output
+Reasoning: First, the question is: "How many r's are in 'strawberry'?"
+
+I need to count the number of times the letter 'r' appears in the word "strawberry".
+
+Let me write out the word: S-T-R-A-W-B-E-R-R-Y.
+
+Now, I'll go through each letter and count the 'r's.
+...
+So, I have three 'r's in "strawberry".
+
+I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.
+
+Therefore, the answer should be 3.
+----------------------------------------------------------------------------------------------------
+Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity:
+
+- The word is spelled: S-T-R-A-W-B-E-R-R-Y
+- The "r" appears at the 3rd, 8th, and 9th positions.
+```
+<Note>
+DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.
+</Note>
+
+```python Example
+# Example with logit_bias parameter
+# Note: You need to get the actual token IDs from your tokenizer
+# For demonstration, we'll use some example token IDs
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "Complete this sentence: The weather today is"}
+    ],
+    temperature=0.7,
+    max_tokens=20,
+    logit_bias={
+        "12345": 50,  # Increase likelihood of token ID 12345
+        "67890": -50,  # Decrease likelihood of token ID 67890
+        "11111": 25,  # Slightly increase likelihood of token ID 11111
+    },
+)
+
+print_highlight(f"Response with logit bias: {response.choices[0].message.content}")
+```
+
+### Parameters
+
+The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.
+
+SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor.
+
+```python Example
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {
+            "role": "system",
+            "content": "You are a knowledgeable historian who provides concise responses.",
+        },
+        {"role": "user", "content": "Tell me about ancient Rome"},
+        {
+            "role": "assistant",
+            "content": "Ancient Rome was a civilization centered in Italy.",
+        },
+        {"role": "user", "content": "What were their major achievements?"},
+    ],
+    temperature=0.3,  # Lower temperature for more focused responses
+    max_tokens=128,  # Reasonable length for a concise response
+    top_p=0.95,  # Slightly higher for better fluency
+    presence_penalty=0.2,  # Mild penalty to avoid repetition
+    frequency_penalty=0.2,  # Mild penalty for more natural language
+    n=1,  # Single response is usually more stable
+    seed=42,  # Keep for reproducibility
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+Streaming mode is also supported.
+
+#### Logit Bias Support
+
+The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.
+
+```python Example
+stream = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[{"role": "user", "content": "Say this is a test"}],
+    stream=True,
+)
+for chunk in stream:
+    if chunk.choices[0].delta.content is not None:
+        print(chunk.choices[0].delta.content, end="")
+```
+
+#### Returning Routed Experts (MoE Models)
+
+For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks.
+
+```python Example
+# Example with logit_bias parameter for completions API
+# Note: You need to get the actual token IDs from your tokenizer
+# For demonstration, we'll use some example token IDs
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="The best programming language for AI is",
+    temperature=0.7,
+    max_tokens=20,
+    logit_bias={
+        "12345": 75,  # Strongly favor token ID 12345
+        "67890": -100,  # Completely avoid token ID 67890
+        "11111": -25,  # Slightly discourage token ID 11111
+    },
+)
+
+print_highlight(f"Response with logit bias: {response.choices[0].text}")
+```
+
+## Completions
+
+### Usage
+Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates.
+
+```python Example
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="List 3 countries and their capitals.",
+    temperature=0,
+    max_tokens=64,
+    n=1,
+    stop=None,
+)
+
+print_highlight(f"Response: {response}")
+```
+
+### Parameters
+
+The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.
+
+Here is an example of a detailed completions request:
+
+```python Example
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="Write a short story about a space explorer.",
+    temperature=0.7,  # Moderate temperature for creative writing
+    max_tokens=150,  # Longer response for a story
+    top_p=0.9,  # Balanced diversity in word choice
+    stop=["\n\n", "THE END"],  # Multiple stop sequences
+    presence_penalty=0.3,  # Encourage novel elements
+    frequency_penalty=0.3,  # Reduce repetitive phrases
+    n=1,  # Generate one completion
+    seed=123,  # For reproducible results
+)
+
+print_highlight(f"Response: {response}")
+```
+
+#### Returning Routed Experts (MoE Models)
+
+For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks.
+
+## Structured Outputs (JSON, Regex, EBNF)
+
+For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details.
+
+## Using LoRA Adapters
+
+SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.
+
+**Server Setup:**
+```bash Command
+python -m sglang.launch_server \
+    --model-path qwen/qwen2.5-0.5b-instruct \
+    --enable-lora \
+    --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b
+```
+
+For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora).
+
+**API Call:**
+
+(Recommended) Use the `model:adapter` syntax to specify which adapter to use:
+```python Example
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct:adapter_a",  # ← base-model:adapter-name
+    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
+    max_tokens=50,
+)
+```
+
+**Backward Compatible: Using `extra_body`**
+
+The old `extra_body` method is still supported for backward compatibility:
+```python Example
+# Backward compatible method
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
+    extra_body={"lora_path": "adapter_a"},  # ← old method
+    max_tokens=50,
+)
+```
+**Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence.
+
+```python Example
+terminate_process(server_process)
+```

docs_new/docs/basic_usage/openai_api_embeddings.ipynb

@@ -0,0 +1,193 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Embedding\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).\n",
+    "\n",
+    "This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/retrieval_ranking/embedding_models.md)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "embedding_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
+    "    --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, json\n",
+    "\n",
+    "text = \"Once upon a time\"\n",
+    "\n",
+    "curl_text = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": \"{text}\"}}'\"\"\"\n",
+    "\n",
+    "result = subprocess.check_output(curl_text, shell=True)\n",
+    "\n",
+    "print(result)\n",
+    "\n",
+    "text_embedding = json.loads(result)[\"data\"][0][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "text = \"Once upon a time\"\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/v1/embeddings\",\n",
+    "    json={\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": text},\n",
+    ")\n",
+    "\n",
+    "text_embedding = response.json()[\"data\"][0][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "# Text embedding example\n",
+    "response = client.embeddings.create(\n",
+    "    model=\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\",\n",
+    "    input=text,\n",
+    ")\n",
+    "\n",
+    "embedding = response.data[0].embedding[:10]\n",
+    "print_highlight(f\"Text embedding (first 10): {embedding}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Input IDs\n",
+    "\n",
+    "SGLang also supports `input_ids` as input to get the embedding."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\")\n",
+    "input_ids = tokenizer.encode(text)\n",
+    "\n",
+    "curl_ids = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": {json.dumps(input_ids)}}}'\"\"\"\n",
+    "\n",
+    "input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[\"data\"][\n",
+    "    0\n",
+    "][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Input IDs embedding (first 10): {input_ids_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multi-Modal Embedding Model\n",
+    "Please refer to [Multi-Modal Embedding Model](../supported_models/retrieval_ranking/embedding_models.md)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_embeddings.mdx

@@ -0,0 +1,126 @@
+---
+title: "OpenAI APIs - Embedding"
+metatags:
+    description: "This tutorial covers the embedding APIs for embedding models."
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).
+
+This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models)
+
+
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
+
+
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+embedding_process, port = launch_server_cmd(
+    """
+python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
+    --host 0.0.0.0 --is-embedding --log-level warning
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+## Using cURL
+
+
+
+```python Example
+import subprocess, json
+
+text = "Once upon a time"
+
+curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""
+
+result = subprocess.check_output(curl_text, shell=True)
+
+print(result)
+
+text_embedding = json.loads(result)["data"][0]["embedding"]
+
+print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
+```
+
+## Using Python Requests
+
+
+
+```python Example
+import requests
+
+text = "Once upon a time"
+
+response = requests.post(
+    f"http://localhost:{port}/v1/embeddings",
+    json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
+)
+
+text_embedding = response.json()["data"][0]["embedding"]
+
+print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
+```
+
+## Using OpenAI Python Client
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+# Text embedding example
+response = client.embeddings.create(
+    model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
+    input=text,
+)
+
+embedding = response.data[0].embedding[:10]
+print_highlight(f"Text embedding (first 10): {embedding}")
+```
+
+## Using Input IDs
+
+SGLang also supports `input_ids` as input to get the embedding.
+
+
+
+```python Example
+import json
+import os
+from transformers import AutoTokenizer
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
+input_ids = tokenizer.encode(text)
+
+curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""
+
+input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
+    0
+]["embedding"]
+
+print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
+```
+
+
+```python Example
+terminate_process(embedding_process)
+```
+
+## Multi-Modal Embedding Model
+Please refer to [Multi-Modal Embedding Model](../supported-models)

docs_new/docs/basic_usage/openai_api_vision.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Vision\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).\n",
+    "This tutorial covers the vision APIs for vision language models.\n",
+    "\n",
+    "SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/text_generation/multimodal_language_models.md).\n",
+    "\n",
+    "As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "example_image_url = \"https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png\"\n",
+    "logo_image_url = (\n",
+    "    \"https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png\"\n",
+    ")\n",
+    "\n",
+    "vision_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=vision_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL\n",
+    "\n",
+    "Once the server is up, you can send test requests using curl or requests."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "\n",
+    "curl_command = f\"\"\"\n",
+    "curl -s http://localhost:{port}/v1/chat/completions \\\\\n",
+    "  -H \"Content-Type: application/json\" \\\\\n",
+    "  -d '{{\n",
+    "    \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    \"messages\": [\n",
+    "      {{\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": [\n",
+    "          {{\n",
+    "            \"type\": \"text\",\n",
+    "            \"text\": \"What’s in this image?\"\n",
+    "          }},\n",
+    "          {{\n",
+    "            \"type\": \"image_url\",\n",
+    "            \"image_url\": {{\n",
+    "              \"url\": \"{example_image_url}\"\n",
+    "            }}\n",
+    "          }}\n",
+    "        ]\n",
+    "      }}\n",
+    "    ],\n",
+    "    \"max_tokens\": 300\n",
+    "  }}'\n",
+    "\"\"\"\n",
+    "\n",
+    "response = subprocess.check_output(curl_command, shell=True).decode()\n",
+    "print_highlight(response)\n",
+    "\n",
+    "\n",
+    "response = subprocess.check_output(curl_command, shell=True).decode()\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/chat/completions\"\n",
+    "\n",
+    "data = {\n",
+    "    \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    \"messages\": [\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\"type\": \"text\", \"text\": \"What’s in this image?\"},\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\"url\": example_image_url},\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    \"max_tokens\": 300,\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                    \"type\": \"text\",\n",
+    "                    \"text\": \"What is in this image?\",\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\"url\": example_image_url},\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    max_tokens=300,\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multiple-Image Inputs\n",
+    "\n",
+    "The server also supports multiple images and interleaved text and images if the model supports it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\n",
+    "                        \"url\": example_image_url,\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\n",
+    "                        \"url\": logo_image_url,\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"text\",\n",
+    "                    \"text\": \"I have two very different images. They are not related at all. \"\n",
+    "                    \"Please describe the first image in one sentence, and then describe the second image in another sentence.\",\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(vision_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_vision.mdx

@@ -0,0 +1,176 @@
+---
+title: "OpenAI APIs - Vision"
+metatags:
+    description: "This tutorial covers the vision APIs for vision language models."
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).
+This tutorial covers the vision APIs for vision language models.
+
+SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models/multimodal_language_models).
+
+As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py).
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize.
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
+logo_image_url = (
+    "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"
+)
+
+vision_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=vision_process)
+```
+
+## Using cURL
+
+Once the server is up, you can send test requests using curl or requests.
+
+```python Example
+import subprocess
+
+curl_command = f"""
+curl -s http://localhost:{port}/v1/chat/completions \\
+  -H "Content-Type: application/json" \\
+  -d '{{
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+      {{
+        "role": "user",
+        "content": [
+          {{
+            "type": "text",
+            "text": "What’s in this image?"
+          }},
+          {{
+            "type": "image_url",
+            "image_url": {{
+              "url": "{example_image_url}"
+            }}
+          }}
+        ]
+      }}
+    ],
+    "max_tokens": 300
+  }}'
+"""
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print_highlight(response)
+
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print_highlight(response)
+```
+
+## Using Python Requests
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/v1/chat/completions"
+
+data = {
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "What’s in this image?"},
+                {
+                    "type": "image_url",
+                    "image_url": {"url": example_image_url},
+                },
+            ],
+        }
+    ],
+    "max_tokens": 300,
+}
+
+response = requests.post(url, json=data)
+print_highlight(response.text)
+```
+
+## Using OpenAI Python Client
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "What is in this image?",
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {"url": example_image_url},
+                },
+            ],
+        }
+    ],
+    max_tokens=300,
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+## Multiple-Image Inputs
+
+The server also supports multiple images and interleaved text and images if the model supports it.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": example_image_url,
+                    },
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": logo_image_url,
+                    },
+                },
+                {
+                    "type": "text",
+                    "text": "I have two very different images. They are not related at all. "
+                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
+                },
+            ],
+        }
+    ],
+    temperature=0,
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+```python Example
+terminate_process(vision_process)
+```

docs_new/docs/basic_usage/overview.mdx

@@ -0,0 +1,12 @@
+---
+title: Basic Usage
+description: Core APIs and common usage patterns for SGLang.
+---
+
+- [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings
+- [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration
+- [Ollama API](./ollama_api)
+- [Offline Engine API](./offline_engine_api)
+- [Native API](./native_api)
+- [Sampling Parameters](./sampling_params)
+- [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more

docs_new/docs/basic_usage/sampling_params.mdx

@@ -0,0 +1,581 @@
+---
+title: "Sampling Parameters"
+metatags:
+    description: "Complete reference for SGLang sampling parameters: temperature, top_p, top_k, frequency penalty, stop tokens, and more."
+---
+This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime.
+If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions).
+
+## `/generate` Endpoint
+
+The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>text</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The input prompt. Can be a single prompt or a batch of prompts.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_ids</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The token IDs for text; one can specify either text or input_ids.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_embeds</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[List[float]]], List[List[float]]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The embeddings for input_ids; one can specify either text, input_ids, or input_embeds.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>image_data</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal-input-formats) for details.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>audio_data</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[AudioDataItem], AudioDataItem]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The audio input. Can be a file name, URL, or base64 encoded string.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>sampling_params</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Dict], Dict]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The sampling parameters as described in the sections below.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rid</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The request ID.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_logprob</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[bool], bool]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return log probabilities for tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>logprob_start_len</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_logprobs_num</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the number of top logprobs to return at each position.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>token_ids_logprob</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the token IDs to return logprob for.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_text_in_logprobs</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to detokenize tokens in text in the returned logprobs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stream</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to stream output.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lora_path</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], Optional[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The path to the LoRA.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_logit_processor</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_hidden_states</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Union[List[bool], bool] = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return hidden states.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_routed_experts</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. With the default `routed_experts_start_len=0`, returns the full available sequence `[0, seqlen - 1)` because RL workflows need routed experts for the full sequence. The result is base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>routed_experts_start_len</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If `return_routed_experts`, the absolute start position for returned routed-experts rows. `0` preserves the default full sequence; set it to an accumulated prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. Must be in `[0, prompt_tokens]`.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Sampling parameters
+
+The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs.
+
+### Note on defaults
+
+By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`.
+
+```bash Command
+# Use model-provided defaults from generation_config.json (default behavior)
+python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model
+
+# Use SGLang/OpenAI constant defaults instead
+python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
+```
+
+### Core parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>max_new_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 128`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The maximum output length measured in tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_token_ids</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_regex</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Stop when hitting any of the regex patterns in this list</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>temperature</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_p</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_k</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int (model default; fallback -1)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_p</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 0.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Penalizers
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>frequency_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>presence_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>repetition_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 1.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values < 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_new_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Constrained decoding
+
+Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>json_schema</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>JSON schema for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>regex</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Regex for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ebnf</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>EBNF for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>structural_tag</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The structal tag for structured outputs.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Other options
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>n</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 1`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ignore_eos</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't stop generation when EOS token is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>skip_special_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Remove special tokens during decoding.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>spaces_between_special_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether or not to add spaces between special tokens during detokenization.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>no_stop_trim</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't trim stop words or EOS token from the generated text.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_params</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[Optional[Dict[str, Any]]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Used when employing `CustomLogitProcessor`. For usage, see below.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Examples
+
+### Normal
+
+Launch a server:
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+print(response.json())
+```
+
+Detailed example in [send request](./send_request).
+
+### Streaming
+
+Send a request and stream the output:
+
+```python Example
+import requests, json
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+        "stream": True,
+    },
+    stream=True,
+)
+
+prev = 0
+for chunk in response.iter_lines(decode_unicode=False):
+    chunk = chunk.decode("utf-8")
+    if chunk and chunk.startswith("data:"):
+        if chunk == "data: [DONE]":
+            break
+        data = json.loads(chunk[5:].strip("\n"))
+        output = data["text"].strip()
+        print(output[prev:], end="", flush=True)
+        prev = len(output)
+print("")
+```
+
+Detailed example in [openai compatible api](./openai_api_completions).
+
+### Multimodal
+
+Launch a server:
+
+```bash Command
+python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov
+```
+
+Download an image:
+
+```bash Command
+curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
+                "<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
+                "<|im_start|>assistant\n",
+        "image_data": "example_image.png",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+print(response.json())
+```
+
+The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`.
+
+Streaming is supported in a similar manner as [above](#streaming).
+
+Detailed example in [OpenAI API Vision](./openai_api_vision).
+
+### Structured Outputs (JSON, Regex, EBNF)
+
+You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.
+
+SGLang supports two grammar backends:
+
+- [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints.
+  - XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README).
+- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
+
+If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag:
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+--port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar)
+```
+
+```python Example
+import json
+import requests
+
+json_schema = json.dumps({
+    "type": "object",
+    "properties": {
+        "name": {"type": "string", "pattern": "^[\\w]+$"},
+        "population": {"type": "integer"},
+    },
+    "required": ["name", "population"],
+})
+
+# JSON (works with both Outlines and XGrammar)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Here is the information of the capital of France in the JSON format.\n",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "json_schema": json_schema,
+        },
+    },
+)
+print(response.json())
+
+# Regular expression (Outlines backend only)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Paris is the capital of",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "regex": "(France|England)",
+        },
+    },
+)
+print(response.json())
+
+# EBNF (XGrammar backend only)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Write a greeting.",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "ebnf": 'root ::= "Hello" | "Hi" | "Hey"',
+        },
+    },
+)
+print(response.json())
+```
+
+Detailed example in [structured outputs](../advanced_features/structured_outputs).
+
+### Custom logit processor
+
+Launch a server with `--enable-custom-logit-processor` flag on.
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+  --port 30000 \
+  --enable-custom-logit-processor
+```
+
+Define a custom logit processor that will always sample a specific token id.
+
+```python Example
+from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor
+
+class DeterministicLogitProcessor(CustomLogitProcessor):
+    """A dummy logit processor that changes the logits to always
+    sample the given token id.
+    """
+
+    def __call__(self, logits, custom_param_list):
+        # Check that the number of logits matches the number of custom parameters
+        assert logits.shape[0] == len(custom_param_list)
+        key = "token_id"
+
+        for i, param_dict in enumerate(custom_param_list):
+            # Mask all other tokens
+            logits[i, :] = -float("inf")
+            # Assign highest probability to the specified token
+            logits[i, param_dict[key]] = 0.0
+        return logits
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
+        "sampling_params": {
+            "temperature": 0.0,
+            "max_new_tokens": 32,
+            "custom_params": {"token_id": 5},
+        },
+    },
+)
+print(response.json())
+```
+
+Send an OpenAI chat completion request:
+
+```python Example
+import openai
+from sglang.utils import print_highlight
+
+client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3-8B-Instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0.0,
+    max_tokens=32,
+    extra_body={
+        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
+        "custom_params": {"token_id": 5},
+    },
+)
+
+print_highlight(f"Response: {response}")
+```

docs_new/docs/basic_usage/send_request.ipynb

@@ -0,0 +1,251 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Sending Requests\n",
+    "This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.\n",
+    "\n",
+    "- For Vision Language Models, see [OpenAI APIs - Vision](openai_api_vision.ipynb).\n",
+    "- For Embedding Models, see [OpenAI APIs - Embedding](openai_api_embeddings.ipynb) and [Encode (embedding model)](native_api.html#Encode-(embedding-model)).\n",
+    "- For Reward Models, see [Classify (reward model)](native_api.html#Classify-(reward-model))."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "# This is equivalent to running the following command in your terminal\n",
+    "# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    " --host 0.0.0.0 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, json\n",
+    "\n",
+    "curl_command = f\"\"\"\n",
+    "curl -s http://localhost:{port}/v1/chat/completions \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"qwen/qwen2.5-0.5b-instruct\", \"messages\": [{{\"role\": \"user\", \"content\": \"What is the capital of France?\"}}]}}'\n",
+    "\"\"\"\n",
+    "\n",
+    "response = json.loads(subprocess.check_output(curl_command, shell=True))\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/chat/completions\"\n",
+    "\n",
+    "data = {\n",
+    "    \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "# Use stream=True for streaming responses\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "# Handle the streaming output\n",
+    "for chunk in response:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        print(chunk.choices[0].delta.content, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Native Generation APIs\n",
+    "\n",
+    "You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](sampling_params.md)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"The capital of France is\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 32,\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests, json\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"The capital of France is\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 32,\n",
+    "        },\n",
+    "        \"stream\": True,\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "prev = 0\n",
+    "for chunk in response.iter_lines(decode_unicode=False):\n",
+    "    chunk = chunk.decode(\"utf-8\")\n",
+    "    if chunk and chunk.startswith(\"data:\"):\n",
+    "        if chunk == \"data: [DONE]\":\n",
+    "            break\n",
+    "        data = json.loads(chunk[5:].strip(\"\\n\"))\n",
+    "        output = data[\"text\"]\n",
+    "        print(output[prev:], end=\"\", flush=True)\n",
+    "        prev = len(output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/send_request.mdx

@@ -0,0 +1,172 @@
+---
+title: "Tutorial: Sending a request"
+metatags:
+    description: "This notebook provides a quick-start guide to use SGLang in chat completions after installation. "
+---
+This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.
+
+- For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision).
+- For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model).
+- For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model).
+
+
+## Launch A Server
+
+
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, terminate_process
+
+# This is equivalent to running the following command in your terminal
+# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0
+
+server_process, port = launch_server_cmd(
+    """
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
+ --host 0.0.0.0 --log-level warning
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+## Using cURL
+
+
+
+
+```python Example
+import subprocess, json
+
+curl_command = f"""
+curl -s http://localhost:{port}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}'
+"""
+
+response = json.loads(subprocess.check_output(curl_command, shell=True))
+print(response)
+```
+
+## Using Python Requests
+
+
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/v1/chat/completions"
+
+data = {
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "messages": [{"role": "user", "content": "What is the capital of France?"}],
+}
+
+response = requests.post(url, json=data)
+print(response.json())
+```
+
+## Using OpenAI Python Client
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+)
+print(response)
+```
+
+### Streaming
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+# Use stream=True for streaming responses
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+    stream=True,
+)
+
+# Handle the streaming output
+for chunk in response:
+    if chunk.choices[0].delta.content:
+        print(chunk.choices[0].delta.content, end="", flush=True)
+```
+
+## Using Native Generation APIs
+
+You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params).
+
+
+
+```python Example
+import requests
+
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+
+print(response.json())
+```
+### Streaming
+
+
+
+```python Example
+import requests, json
+
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+        "stream": True,
+    },
+    stream=True,
+)
+
+prev = 0
+for chunk in response.iter_lines(decode_unicode=False):
+    chunk = chunk.decode("utf-8")
+    if chunk and chunk.startswith("data:"):
+        if chunk == "data: [DONE]":
+            break
+        data = json.loads(chunk[5:].strip("\n"))
+        output = data["text"]
+        print(output[prev:], end="", flush=True)
+        prev = len(output)
+```
+
+```python Example
+terminate_process(server_process)
+```