chore: import upstream snapshot with attribution

docs_new/docs/references/custom_chat_template.mdx

@@ -0,0 +1,54 @@
+---
+title: "Custom Chat Template"
+metatags:
+    description: "SGLang custom chat templates: JSON and Jinja formats for OpenAI-compatible API server. Override tokenizer defaults."
+---
+**NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)).
+
+By default, the server uses the chat template specified in the model tokenizer from Hugging Face.
+It should just work for most official models such as Llama-2/Llama-3.
+
+If needed, you can also override the chat template when launching the server:
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-2-7b-chat-hf \
+  --port 30000 \
+  --chat-template llama-2
+```
+
+If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file.
+
+## JSON Format
+
+You can load the JSON format, which is defined by `conversation.py`.
+
+```json Config
+{
+  "name": "my_model",
+  "system": "<|im_start|>system",
+  "user": "<|im_start|>user",
+  "assistant": "<|im_start|>assistant",
+  "sep_style": "CHATML",
+  "sep": "<|im_end|>",
+  "stop_str": ["<|im_end|>", "<|im_start|>"]
+}
+```
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-2-7b-chat-hf \
+  --port 30000 \
+  --chat-template ./my_model_template.json
+```
+
+## Jinja Format
+
+You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers.
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-2-7b-chat-hf \
+  --port 30000 \
+  --chat-template ./my_model_template.jinja
+```

docs_new/docs/references/faq.mdx

@@ -0,0 +1,42 @@
+---
+title: "Troubleshooting and Frequently Asked Questions"
+metatags:
+    description: "SGLang troubleshooting: CUDA OOM, illegal memory access, server hangs, non-deterministic outputs."
+---
+## Troubleshooting
+
+This page lists common errors and tips for resolving them.
+
+### CUDA Out of Memory
+If you encounter out-of-memory (OOM) errors, you can adjust the following parameters:
+
+- If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts.
+- If OOM occurs during decoding, try lowering `--max-running-requests`.
+- You can also decrease `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
+- Another common case for OOM is requesting input logprobs for a long prompt as it requires significant memory. To address this, set `logprob_start_len` in your sampling parameters to include only the necessary parts. If you do need input logprobs for a long prompt, try reducing `--mem-fraction-static`.
+
+### CUDA Error: Illegal Memory Access Encountered
+This error may result from kernel errors or out-of-memory issues:
+- If it is a kernel error, resolving it may be challenging. Please file an issue on GitHub.
+- If it is an out-of-memory issue, it may sometimes be reported as this error instead of "Out of Memory." Refer to the section above for guidance on avoiding OOM issues.
+
+### The server hangs
+- If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang.
+    - If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case,
+      you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs-decode`, or decrease `--chunked-prefill-size`.
+- Other bugs, please file an issue on GitHub.
+
+
+## Frequently Asked Questions
+
+### The results are not deterministic, even with a temperature of 0
+
+You may notice that when you send the same request twice, the results from the engine will be slightly different, even when the temperature is set to 0.
+
+From our initial investigation, this indeterminism arises from two factors: dynamic batching and prefix caching. Roughly speaking, dynamic batching accounts for about 95% of the indeterminism, while prefix caching accounts for the remaining portion. The server runs dynamic batching under the hood. Different batch sizes can cause PyTorch/CuBLAS to dispatch to different CUDA kernels, which can lead to slight numerical differences. This difference accumulates across many layers, resulting in nondeterministic output when the batch size changes. Similarly, when prefix caching is enabled, it can also dispatch to different kernels. Even when the computations are mathematically equivalent, small numerical differences from different kernel implementations lead to the final nondeterministic outputs.
+
+To achieve more deterministic outputs in the current code, you can add `--disable-radix-cache` and send only one request at a time. The results will be mostly deterministic under this setting.
+
+**Update**:
+Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`.
+Please find more details in this blog post: https://lmsys.org/blog/2025-09-22-sglang-deterministic/

docs_new/docs/references/frontend/choices_methods.mdx

@@ -0,0 +1,81 @@
+---
+title: "Choices Methods in SGLang"
+metatags:
+    description: "SGLang choices methods: token_length_normalized, greedy_token_selection, unconditional_likelihood_normalized."
+---
+This doc describes the choices methods supported by SGLang.
+
+The optional `choices_method` arg determines how options supplied to SGLang's `choices` primitive are selected. Only the `RuntimeEndpoint` backend supports the `choices_method` arg. Other backends, such as `OpenAI`, have bespoke selection implementations due to API limitations.
+
+## Methods
+
+### Token Length Normalized
+
+Token length normalized is the default SGLang choices method. It selects the option with the highest average logprob across all of its tokens.
+
+Usage example (alternatively, simply omit the `choices_method` arg):
+```python Example
+@sgl.function
+def example(s):
+    s += sgl.user("What is the capital of France?")
+    s += sgl.assistant(
+        sgl.gen(
+            "answer",
+            choices=["London", "Paris", "Berlin"],
+            choices_method=sgl.token_length_normalized,
+        )
+    )
+```
+
+
+This can perform poorly if an option contains many tokens, where its later tokens are predicted with high confidence based on its earlier tokens. For instance, even strong models will fail the above example if the specified options are `["Paris", "Antidisestablishmentarianism"]`.
+
+### Greedy Token Selection
+
+Greedy token selection simply selects the option with the highest logprob for its initial token. For overlapping options where one option is a subset of a longer option, the logprobs of the shorter option are extended using its average logprob for comparison against the longer option.
+
+Usage example:
+```python Example
+@sgl.function
+def example(s):
+    s += sgl.user("What is the capital of France?")
+    s += sgl.assistant(
+        sgl.gen(
+            "answer",
+            choices=["London", "Paris", "Berlin"],
+            choices_method=sgl.greedy_token_selection,
+        )
+    )
+```
+
+This can perform poorly if an option misleads the model down a bad path based on an attractive initial token. For instance, greedy selection will result in an incorrect response for this example:
+```python Example
+@sgl.function
+def us_president_example(s):
+    s += sgl.user("Name a US president.")
+    s += sgl.assistant(
+        sgl.gen(
+            "answer",
+            choices=["Donald Duck", "Millard Fillmore"],
+            choices_method=sgl.greedy_token_selection,
+        )
+    )
+```
+
+### Unconditional Likelihood Normalized
+
+Unconditional likelihood normalized selects the option with the highest average token logprob once normalized by the unconditional token logprobs, as described in [this EleutherAI blogpost](https://blog.eleuther.ai/multiple-choice-normalization/). This method incurs an additional LLM call to obtain the unconditional likelihoods.
+
+Usage example:
+```python Example
+@sgl.function
+def example(s):
+    s += sgl.user("What is the capital of France?")
+    s += sgl.assistant(
+        sgl.gen(
+            "answer",
+            choices=["London", "Paris", "Berlin"],
+            choices_method=sgl.unconditional_likelihood_normalized,
+        )
+    )
+```

docs_new/docs/references/frontend/frontend_index.mdx

@@ -0,0 +1,7 @@
+---
+title: "Frontend Language"
+metatags:
+    description: "SGLang frontend language documentation: tutorials and choices methods reference."
+---
+- [Frontend Tutorial](./frontend_tutorial)
+- [Choices Methods](./choices_methods)

docs_new/docs/references/frontend/frontend_index.rst

@@ -0,0 +1,9 @@
+Frontend Language
+=================
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Frontend Language
+
+   frontend_tutorial.ipynb
+   choices_methods.md

docs_new/docs/references/frontend/frontend_tutorial.ipynb

@@ -0,0 +1,456 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SGLang Frontend Language"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang import assistant_begin, assistant_end\n",
+    "from sglang import assistant, function, gen, system, user\n",
+    "from sglang import image\n",
+    "from sglang import RuntimeEndpoint\n",
+    "from sglang.lang.api import set_default_backend\n",
+    "from sglang.srt.utils import load_image\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import print_highlight, terminate_process, wait_for_server\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "print(f\"Server started on http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Set the default backend. Note: Besides the local server, you may use also `OpenAI` or other API endpoints."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "set_default_backend(RuntimeEndpoint(f\"http://localhost:{port}\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Usage\n",
+    "\n",
+    "The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def basic_qa(s, question):\n",
+    "    s += system(f\"You are a helpful assistant than can answer questions.\")\n",
+    "    s += user(question)\n",
+    "    s += assistant(gen(\"answer\", max_tokens=512))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "state = basic_qa(\"List 3 countries and their capitals.\")\n",
+    "print_highlight(state[\"answer\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multi-turn Dialog\n",
+    "\n",
+    "SGLang frontend language can also be used to define multi-turn dialogs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def multi_turn_qa(s):\n",
+    "    s += system(f\"You are a helpful assistant than can answer questions.\")\n",
+    "    s += user(\"Please give me a list of 3 countries and their capitals.\")\n",
+    "    s += assistant(gen(\"first_answer\", max_tokens=512))\n",
+    "    s += user(\"Please give me another list of 3 countries and their capitals.\")\n",
+    "    s += assistant(gen(\"second_answer\", max_tokens=512))\n",
+    "    return s\n",
+    "\n",
+    "\n",
+    "state = multi_turn_qa()\n",
+    "print_highlight(state[\"first_answer\"])\n",
+    "print_highlight(state[\"second_answer\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Control flow\n",
+    "\n",
+    "You may use any Python code within the function to define more complex control flows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def tool_use(s, question):\n",
+    "    s += assistant(\n",
+    "        \"To answer this question: \"\n",
+    "        + question\n",
+    "        + \". I need to use a \"\n",
+    "        + gen(\"tool\", choices=[\"calculator\", \"search engine\"])\n",
+    "        + \". \"\n",
+    "    )\n",
+    "\n",
+    "    if s[\"tool\"] == \"calculator\":\n",
+    "        s += assistant(\"The math expression is: \" + gen(\"expression\"))\n",
+    "    elif s[\"tool\"] == \"search engine\":\n",
+    "        s += assistant(\"The key word to search is: \" + gen(\"word\"))\n",
+    "\n",
+    "\n",
+    "state = tool_use(\"What is 2 * 2?\")\n",
+    "print_highlight(state[\"tool\"])\n",
+    "print_highlight(state[\"expression\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Parallelism\n",
+    "\n",
+    "Use `fork` to launch parallel prompts. Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def tip_suggestion(s):\n",
+    "    s += assistant(\n",
+    "        \"Here are two tips for staying healthy: \"\n",
+    "        \"1. Balanced Diet. 2. Regular Exercise.\\n\\n\"\n",
+    "    )\n",
+    "\n",
+    "    forks = s.fork(2)\n",
+    "    for i, f in enumerate(forks):\n",
+    "        f += assistant(\n",
+    "            f\"Now, expand tip {i+1} into a paragraph:\\n\"\n",
+    "            + gen(\"detailed_tip\", max_tokens=256, stop=\"\\n\\n\")\n",
+    "        )\n",
+    "\n",
+    "    s += assistant(\"Tip 1:\" + forks[0][\"detailed_tip\"] + \"\\n\")\n",
+    "    s += assistant(\"Tip 2:\" + forks[1][\"detailed_tip\"] + \"\\n\")\n",
+    "    s += assistant(\n",
+    "        \"To summarize the above two tips, I can say:\\n\" + gen(\"summary\", max_tokens=512)\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "state = tip_suggestion()\n",
+    "print_highlight(state[\"summary\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Constrained Decoding\n",
+    "\n",
+    "Use `regex` to specify a regular expression as a decoding constraint. This is only supported for local models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def regular_expression_gen(s):\n",
+    "    s += user(\"What is the IP address of the Google DNS servers?\")\n",
+    "    s += assistant(\n",
+    "        gen(\n",
+    "            \"answer\",\n",
+    "            temperature=0,\n",
+    "            regex=r\"((25[0-5]|2[0-4]\\d|[01]?\\d\\d?).){3}(25[0-5]|2[0-4]\\d|[01]?\\d\\d?)\",\n",
+    "        )\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "state = regular_expression_gen()\n",
+    "print_highlight(state[\"answer\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use `regex` to define a `JSON` decoding schema."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "character_regex = (\n",
+    "    r\"\"\"\\{\\n\"\"\"\n",
+    "    + r\"\"\"    \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"house\": \"(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"blood status\": \"(Pure-blood|Half-blood|Muggle-born)\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"occupation\": \"(student|teacher|auror|ministry of magic|death eater|order of the phoenix)\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"wand\": \\{\\n\"\"\"\n",
+    "    + r\"\"\"        \"wood\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n",
+    "    + r\"\"\"        \"core\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n",
+    "    + r\"\"\"        \"length\": [0-9]{1,2}\\.[0-9]{0,2}\\n\"\"\"\n",
+    "    + r\"\"\"    \\},\\n\"\"\"\n",
+    "    + r\"\"\"    \"alive\": \"(Alive|Deceased)\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"patronus\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n",
+    "    + r\"\"\"    \"bogart\": \"[\\w\\d\\s]{1,16}\"\\n\"\"\"\n",
+    "    + r\"\"\"\\}\"\"\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "@function\n",
+    "def character_gen(s, name):\n",
+    "    s += user(\n",
+    "        f\"{name} is a character in Harry Potter. Please fill in the following information about this character.\"\n",
+    "    )\n",
+    "    s += assistant(gen(\"json_output\", max_tokens=256, regex=character_regex))\n",
+    "\n",
+    "\n",
+    "state = character_gen(\"Harry Potter\")\n",
+    "print_highlight(state[\"json_output\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Batching \n",
+    "\n",
+    "Use `run_batch` to run a batch of prompts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def text_qa(s, question):\n",
+    "    s += user(question)\n",
+    "    s += assistant(gen(\"answer\", stop=\"\\n\"))\n",
+    "\n",
+    "\n",
+    "states = text_qa.run_batch(\n",
+    "    [\n",
+    "        {\"question\": \"What is the capital of the United Kingdom?\"},\n",
+    "        {\"question\": \"What is the capital of France?\"},\n",
+    "        {\"question\": \"What is the capital of Japan?\"},\n",
+    "    ],\n",
+    "    progress_bar=True,\n",
+    ")\n",
+    "\n",
+    "for i, state in enumerate(states):\n",
+    "    print_highlight(f\"Answer {i+1}: {states[i]['answer']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Streaming \n",
+    "\n",
+    "Use `stream` to stream the output to the user."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def text_qa(s, question):\n",
+    "    s += user(question)\n",
+    "    s += assistant(gen(\"answer\", stop=\"\\n\"))\n",
+    "\n",
+    "\n",
+    "state = text_qa.run(\n",
+    "    question=\"What is the capital of France?\", temperature=0.1, stream=True\n",
+    ")\n",
+    "\n",
+    "for out in state.text_iter():\n",
+    "    print(out, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Complex Prompts\n",
+    "\n",
+    "You may use `{system|user|assistant}_{begin|end}` to define complex prompts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def chat_example(s):\n",
+    "    s += system(\"You are a helpful assistant.\")\n",
+    "    # Same as: s += s.system(\"You are a helpful assistant.\")\n",
+    "\n",
+    "    with s.user():\n",
+    "        s += \"Question: What is the capital of France?\"\n",
+    "\n",
+    "    s += assistant_begin()\n",
+    "    s += \"Answer: \" + gen(\"answer\", max_tokens=100, stop=\"\\n\")\n",
+    "    s += assistant_end()\n",
+    "\n",
+    "\n",
+    "state = chat_example()\n",
+    "print_highlight(state[\"answer\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multi-modal Generation\n",
+    "\n",
+    "You may use SGLang frontend language to define multi-modal prompts.\n",
+    "See [here](https://docs.sglang.io/supported_models/text_generation/multimodal_language_models.html) for supported models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "print(f\"Server started on http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "set_default_backend(RuntimeEndpoint(f\"http://localhost:{port}\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Ask a question about an image."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@function\n",
+    "def image_qa(s, image_file, question):\n",
+    "    s += user(image(image_file) + question)\n",
+    "    s += assistant(gen(\"answer\", max_tokens=256))\n",
+    "\n",
+    "\n",
+    "image_url = \"https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png\"\n",
+    "image_bytes, _ = load_image(image_url)\n",
+    "state = image_qa(image_bytes, \"What is in the image?\")\n",
+    "print_highlight(state[\"answer\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/references/frontend/frontend_tutorial.mdx

@@ -0,0 +1,287 @@
+---
+title: "SGLang Frontend Language"
+metatags:
+    description: "SGLang frontend tutorial: multi-turn dialog, fork parallelism, regex constraints, batching, streaming."
+---
+SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way.
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize.
+
+```python Example
+from sglang import assistant_begin, assistant_end
+from sglang import assistant, function, gen, system, user
+from sglang import image
+from sglang import RuntimeEndpoint
+from sglang.lang.api import set_default_backend
+from sglang.srt.utils import load_image
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import print_highlight, terminate_process, wait_for_server
+
+server_process, port = launch_server_cmd(
+    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=server_process)
+print(f"Server started on http://localhost:{port}")
+```
+
+Set the default backend. Note: Besides the local server, you may use also `OpenAI` or other API endpoints.
+
+```python Example
+set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))
+```
+
+## Basic Usage
+
+The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant.
+
+```python Example
+@function
+def basic_qa(s, question):
+    s += system(f"You are a helpful assistant than can answer questions.")
+    s += user(question)
+    s += assistant(gen("answer", max_tokens=512))
+```
+
+```python Example
+state = basic_qa("List 3 countries and their capitals.")
+print_highlight(state["answer"])
+```
+
+## Multi-turn Dialog
+
+SGLang frontend language can also be used to define multi-turn dialogs.
+
+```python Example
+@function
+def multi_turn_qa(s):
+    s += system(f"You are a helpful assistant than can answer questions.")
+    s += user("Please give me a list of 3 countries and their capitals.")
+    s += assistant(gen("first_answer", max_tokens=512))
+    s += user("Please give me another list of 3 countries and their capitals.")
+    s += assistant(gen("second_answer", max_tokens=512))
+    return s
+
+
+state = multi_turn_qa()
+print_highlight(state["first_answer"])
+print_highlight(state["second_answer"])
+```
+
+## Control flow
+
+You may use any Python code within the function to define more complex control flows.
+
+```python Example
+@function
+def tool_use(s, question):
+    s += assistant(
+        "To answer this question: "
+        + question
+        + ". I need to use a "
+        + gen("tool", choices=["calculator", "search engine"])
+        + ". "
+    )
+
+    if s["tool"] == "calculator":
+        s += assistant("The math expression is: " + gen("expression"))
+    elif s["tool"] == "search engine":
+        s += assistant("The key word to search is: " + gen("word"))
+
+
+state = tool_use("What is 2 * 2?")
+print_highlight(state["tool"])
+print_highlight(state["expression"])
+```
+
+## Parallelism
+
+Use `fork` to launch parallel prompts. Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.
+
+```python Example
+@function
+def tip_suggestion(s):
+    s += assistant(
+        "Here are two tips for staying healthy: "
+        "1. Balanced Diet. 2. Regular Exercise.\n\n"
+    )
+
+    forks = s.fork(2)
+    for i, f in enumerate(forks):
+        f += assistant(
+            f"Now, expand tip {i+1} into a paragraph:\n"
+            + gen("detailed_tip", max_tokens=256, stop="\n\n")
+        )
+
+    s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n")
+    s += assistant("Tip 2:" + forks[1]["detailed_tip"] + "\n")
+    s += assistant(
+        "To summarize the above two tips, I can say:\n" + gen("summary", max_tokens=512)
+    )
+
+
+state = tip_suggestion()
+print_highlight(state["summary"])
+```
+
+## Constrained Decoding
+
+Use `regex` to specify a regular expression as a decoding constraint. This is only supported for local models.
+
+```python Example
+@function
+def regular_expression_gen(s):
+    s += user("What is the IP address of the Google DNS servers?")
+    s += assistant(
+        gen(
+            "answer",
+            temperature=0,
+            regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
+        )
+    )
+
+
+state = regular_expression_gen()
+print_highlight(state["answer"])
+```
+
+Use `regex` to define a `JSON` decoding schema.
+
+```python Example
+character_regex = (
+    r"""\{\n"""
+    + r"""    "name": "[\w\d\s]{1,16}",\n"""
+    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
+    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
+    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
+    + r"""    "wand": \{\n"""
+    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
+    + r"""        "core": "[\w\d\s]{1,16}",\n"""
+    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
+    + r"""    \},\n"""
+    + r"""    "alive": "(Alive|Deceased)",\n"""
+    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
+    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
+    + r"""\}"""
+)
+
+
+@function
+def character_gen(s, name):
+    s += user(
+        f"{name} is a character in Harry Potter. Please fill in the following information about this character."
+    )
+    s += assistant(gen("json_output", max_tokens=256, regex=character_regex))
+
+
+state = character_gen("Harry Potter")
+print_highlight(state["json_output"])
+```
+
+## Batching
+
+Use `run_batch` to run a batch of prompts.
+
+```python Example
+@function
+def text_qa(s, question):
+    s += user(question)
+    s += assistant(gen("answer", stop="\n"))
+
+
+states = text_qa.run_batch(
+    [
+        {"question": "What is the capital of the United Kingdom?"},
+        {"question": "What is the capital of France?"},
+        {"question": "What is the capital of Japan?"},
+    ],
+    progress_bar=True,
+)
+
+for i, state in enumerate(states):
+    print_highlight(f"Answer {i+1}: {states[i]['answer']}")
+```
+
+## Streaming
+
+Use `stream` to stream the output to the user.
+
+```python Example
+@function
+def text_qa(s, question):
+    s += user(question)
+    s += assistant(gen("answer", stop="\n"))
+
+
+state = text_qa.run(
+    question="What is the capital of France?", temperature=0.1, stream=True
+)
+
+for out in state.text_iter():
+    print(out, end="", flush=True)
+```
+
+## Complex Prompts
+
+You may use `{system|user|assistant}_{begin|end}` to define complex prompts.
+
+```python Example
+@function
+def chat_example(s):
+    s += system("You are a helpful assistant.")
+    # Same as: s += s.system("You are a helpful assistant.")
+
+    with s.user():
+        s += "Question: What is the capital of France?"
+
+    s += assistant_begin()
+    s += "Answer: " + gen("answer", max_tokens=100, stop="\n")
+    s += assistant_end()
+
+
+state = chat_example()
+print_highlight(state["answer"])
+```
+
+```python Example
+terminate_process(server_process)
+```
+
+## Multi-modal Generation
+
+You may use SGLang frontend language to define multi-modal prompts.
+See [here](../../supported-models/multimodal_language_models) for supported models.
+
+```python Example
+server_process, port = launch_server_cmd(
+    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=server_process)
+print(f"Server started on http://localhost:{port}")
+```
+
+```python Example
+set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))
+```
+
+Ask a question about an image.
+
+```python Example
+@function
+def image_qa(s, image_file, question):
+    s += user(image(image_file) + question)
+    s += assistant(gen("answer", max_tokens=256))
+
+
+image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
+image_bytes, _ = load_image(image_url)
+state = image_qa(image_bytes, "What is in the image?")
+print_highlight(state["answer"])
+```
+
+```python Example
+terminate_process(server_process)
+```

docs_new/docs/references/multi_node_deployment/deploy_on_k8s.mdx

@@ -0,0 +1,340 @@
+---
+title: "Deploy On Kubernetes"
+metatags:
+    description: "SGLang on Kubernetes with LWS: DeepSeek-R1 multi-node deployment, RoCE RDMA setup, NCCL debugging."
+---
+This document is for deploying a RoCE network-based SGLang two-node inference service on a Kubernetes (K8S) cluster.
+
+[LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads. A major use case is for multi-host/multi-node distributed inference.
+
+SGLang can also be deployed with LWS on Kubernetes for distributed model serving.
+
+Please see this guide for more details on deploying SGLang on Kubernetes using LWS.
+
+Here we take the deployment of DeepSeek-R1 as an example.
+
+## Prerequisites
+
+1. At least two Kubernetes nodes, each with two H20 systems and eight GPUs, are required.
+
+2. Make sure your K8S cluster has LWS correctly installed. If it hasn't been set up yet, please follow the [installation instructions](https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index). **Note:** For LWS versions ≤0.5.x, you must use the Downward API to obtain `LWS_WORKER_INDEX`, as native support for this feature was introduced in v0.6.0.
+
+## Basic example
+
+For the basic example documentation, refer to [Deploy Distributed Inference Service with SGLang and LWS on GPUs](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sglang).
+
+However, that document only covers the basic NCCL socket mode.
+
+In this section, we’ll make some simple modifications to adapt the setup to the RDMA scenario.
+
+## RDMA RoCE case
+
+* Check your env:
+
+```bash Command
+[root@node1 ~]# ibstatus
+Infiniband device 'mlx5_bond_0' port 1 status:
+        default gid:     fe80:0000:0000:0000:0225:9dff:fe64:c79a
+        base lid:        0x0
+        sm lid:          0x0
+        state:           4: ACTIVE
+        phys state:      5: LinkUp
+        rate:            200 Gb/sec (2X NDR)
+        link_layer:      Ethernet
+
+Infiniband device 'mlx5_bond_1' port 1 status:
+        default gid:     fe80:0000:0000:0000:0225:9dff:fe6e:c3ec
+        base lid:        0x0
+        sm lid:          0x0
+        state:           4: ACTIVE
+        phys state:      5: LinkUp
+        rate:            200 Gb/sec (2X NDR)
+        link_layer:      Ethernet
+
+Infiniband device 'mlx5_bond_2' port 1 status:
+        default gid:     fe80:0000:0000:0000:0225:9dff:fe73:0dd7
+        base lid:        0x0
+        sm lid:          0x0
+        state:           4: ACTIVE
+        phys state:      5: LinkUp
+        rate:            200 Gb/sec (2X NDR)
+        link_layer:      Ethernet
+
+Infiniband device 'mlx5_bond_3' port 1 status:
+        default gid:     fe80:0000:0000:0000:0225:9dff:fe36:f7ff
+        base lid:        0x0
+        sm lid:          0x0
+        state:           4: ACTIVE
+        phys state:      5: LinkUp
+        rate:            200 Gb/sec (2X NDR)
+        link_layer:      Ethernet
+```
+
+* Prepare the `lws.yaml` file for deploying on k8s.
+
+```yaml Config
+apiVersion: leaderworkerset.x-k8s.io/v1
+kind: LeaderWorkerSet
+metadata:
+  name: sglang
+spec:
+  replicas: 1
+  leaderWorkerTemplate:
+    size: 2
+    restartPolicy: RecreateGroupOnPodRestart
+    leaderTemplate:
+      metadata:
+        labels:
+          role: leader
+      spec:
+        dnsPolicy: ClusterFirstWithHostNet
+        hostNetwork: true
+        hostIPC: true
+        containers:
+          - name: sglang-leader
+            image: sglang:latest
+            securityContext:
+              privileged: true
+            env:
+              - name: NCCL_IB_GID_INDEX
+                value: "3"
+            command:
+              - python3
+              - -m
+              - sglang.launch_server
+              - --model-path
+              - /work/models
+              - --mem-fraction-static
+              -  "0.93"
+              - --torch-compile-max-bs
+              - "8"
+              - --max-running-requests
+              - "20"
+              - --tp
+              - "16" # Size of Tensor Parallelism
+              - --dist-init-addr
+              - $(LWS_LEADER_ADDRESS):20000
+              - --nnodes
+              - $(LWS_GROUP_SIZE)
+              - --node-rank
+              - $(LWS_WORKER_INDEX)
+              - --trust-remote-code
+              - --host
+              - "0.0.0.0"
+              - --port
+              - "40000"
+            resources:
+              limits:
+                nvidia.com/gpu: "8"
+            ports:
+              - containerPort: 40000
+            readinessProbe:
+              tcpSocket:
+                port: 40000
+              initialDelaySeconds: 15
+              periodSeconds: 10
+            volumeMounts:
+              - mountPath: /dev/shm
+                name: dshm
+              - name: model
+                mountPath: /work/models
+              - name: ib
+                mountPath: /dev/infiniband
+        volumes:
+          - name: dshm
+            emptyDir:
+              medium: Memory
+          - name: model
+            hostPath:
+              path: '< your models dir >' # modify it according your models dir
+          - name: ib
+            hostPath:
+              path: /dev/infiniband
+    workerTemplate:
+      spec:
+        dnsPolicy: ClusterFirstWithHostNet
+        hostNetwork: true
+        hostIPC: true
+        containers:
+          - name: sglang-worker
+            image: sglang:latest
+            securityContext:
+              privileged: true
+            env:
+            - name: NCCL_IB_GID_INDEX
+              value: "3"
+            command:
+              - python3
+              - -m
+              - sglang.launch_server
+              - --model-path
+              - /work/models
+              - --mem-fraction-static
+              - "0.93"
+              - --torch-compile-max-bs
+              - "8"
+              - --max-running-requests
+              - "20"
+              - --tp
+              - "16" # Size of Tensor Parallelism
+              - --dist-init-addr
+              - $(LWS_LEADER_ADDRESS):20000
+              - --nnodes
+              - $(LWS_GROUP_SIZE)
+              - --node-rank
+              - $(LWS_WORKER_INDEX)
+              - --trust-remote-code
+            resources:
+              limits:
+                nvidia.com/gpu: "8"
+            volumeMounts:
+              - mountPath: /dev/shm
+                name: dshm
+              - name: model
+                mountPath: /work/models
+              - name: ib
+                mountPath: /dev/infiniband
+        volumes:
+          - name: dshm
+            emptyDir:
+              medium: Memory
+          - name: ib
+            hostPath:
+              path: /dev/infiniband
+          - name: model
+            hostPath:
+              path: /data1/models/deepseek_v3_moe
+***
+apiVersion: v1
+kind: Service
+metadata:
+  name: sglang-leader
+spec:
+  selector:
+    leaderworkerset.sigs.k8s.io/name: sglang
+    role: leader
+  ports:
+    - protocol: TCP
+      port: 40000
+      targetPort: 40000
+
+```
+
+* Then use  `kubectl apply -f lws.yaml` you will get this output.
+
+```text Output
+NAME           READY   STATUS    RESTARTS       AGE
+sglang-0       0/1     Running   0              9s
+sglang-0-1     1/1     Running   0              9s
+```
+
+Wait for the sglang leader (`sglang-0`) status to change to 1/1, which indicates it is `Ready`.
+
+You can use the command `kubectl logs -f sglang-0` to view the logs of the leader node.
+
+Once successful, you should see output like this:
+
+```text Output
+[2025-02-17 05:27:24 TP1] Capture cuda graph end. Time elapsed: 84.89 s
+[2025-02-17 05:27:24 TP6] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP0] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP7] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP3] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP2] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP4] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP1] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24 TP5] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
+[2025-02-17 05:27:24] INFO:     Started server process [1]
+[2025-02-17 05:27:24] INFO:     Waiting for application startup.
+[2025-02-17 05:27:24] INFO:     Application startup complete.
+[2025-02-17 05:27:24] INFO:     Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit)
+[2025-02-17 05:27:25] INFO:     127.0.0.1:48908 - "GET /get_model_info HTTP/1.1" 200 OK
+[2025-02-17 05:27:25 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
+[2025-02-17 05:27:32] INFO:     127.0.0.1:48924 - "POST /generate HTTP/1.1" 200 OK
+[2025-02-17 05:27:32] The server is fired up and ready to roll!
+```
+
+If it doesn’t start up successfully, please follow these steps to check for any remaining issues. Thanks!
+
+### Debug
+
+* Set `NCCL_DEBUG=TRACE` to check if it is a NCCL communication problem.
+
+This should resolve most NCCL-related issues.
+
+***Notice: If you find that NCCL_DEBUG=TRACE is not effective in the container environment, but the process is stuck or you encounter hard-to-diagnose issues, try switching to a different container image. Some images may not handle standard error output properly.***
+
+#### RoCE scenario
+
+* Please make sure that RDMA devices are available in the cluster environment.
+* Please make sure that the nodes in the cluster have Mellanox NICs with RoCE. In this example, we use Mellanox ConnectX 5 model NICs, and the proper OFED driver has been installed. If not, please refer to the document [Install OFED Driver](https://docs.nvidia.com/networking/display/mlnxofedv461000/installing+mellanox+ofed) to install the driver.
+* Check your env:
+
+  ```shell Command
+  $ lspci -nn | grep Eth | grep Mellanox
+  0000:7f:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0000:7f:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0000:c7:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0000:c7:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0001:08:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0001:08:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0001:a2:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  0001:a2:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
+  ```
+
+* Check the OFED driver:
+
+  ```shell Command
+  ofed_info -s
+  OFED-internal-23.07-0.5.0:
+  ```
+
+* Show RDMA link status and check IB devices:
+
+  ```shell Command
+  $ rdma link show
+  8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0
+  9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2
+  10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4
+  11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6
+
+  $ ibdev2netdev
+  8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0
+  9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2
+  10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4
+  11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6
+  ```
+
+* Test RoCE network speed on the host:
+
+  ```shell Command
+  yum install qperf
+  # for server：
+  execute qperf
+  # for client
+  qperf -t 60 -cm1 <server_ip>   rc_rdma_write_bw
+  ```
+
+* Check RDMA accessible in your container:
+
+  ```shell Command
+  # ibv_devices
+  # ibv_devinfo
+  ```
+
+## Keys to success
+
+* In the YAML configuration above, pay attention to the NCCL environment variable. For older versions of NCCL, you should check the NCCL_IB_GID_INDEX environment setting.
+* NCCL_SOCKET_IFNAME is also crucial, but in a containerized environment, this typically isn’t an issue.
+* In some cases, it’s necessary to configure GLOO_SOCKET_IFNAME correctly.
+* NCCL_DEBUG is essential for troubleshooting, but I've found that sometimes it doesn't show error logs within containers. This could be related to the Docker image you're using. You may want to try switching images if needed.
+* Avoid using Docker images based on Ubuntu 18.04, as they tend to have compatibility issues.
+
+## Remaining issues
+
+* In Kubernetes, Docker, or Containerd environments, we use hostNetwork to prevent performance degradation.
+* We utilize privileged mode, which  isn’t secure. Additionally, in containerized environments, full GPU isolation cannot be achieved.
+
+## TODO
+
+* Integrated with [k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin).

docs_new/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy.mdx

@@ -0,0 +1,786 @@
+---
+title: "LWS Based PD Deploy"
+metatags:
+    description: "SGLang LWS PD deployment: DeepSeek R1 prefill/decode disaggregation on Kubernetes with RDMA."
+---
+## 0. Prerequisites
+
+1. k8s >=1.26
+2. lws installed on k8s.
+
+## 1. Image Preparation
+
+`lmsysorg/sglang:deepep`
+
+## 2. Deployment Manifest Files
+
+***Notice: We will package all deployment files into Helm Chart format in the near future. Interested community members can contact us to contribute***
+
+### Prefill
+
+Prefill manifest file [prefill.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p.yaml)
+
+*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*
+
+```yaml Config
+apiVersion: leaderworkerset.x-k8s.io/v1
+kind: LeaderWorkerSet
+metadata:
+  name: deepseekr10528-prefill-main
+spec:
+  leaderWorkerTemplate:
+    leaderTemplate:
+      metadata:
+        labels:
+          role: leader
+      spec:
+        containers:
+        - command:
+          - python3
+          - -m
+          - sglang.launch_server
+          - --port
+          - "30000"
+          - --host
+          - "0.0.0.0"
+          - --model-path
+          - /work/models
+          - --disaggregation-ib-device
+          # should modify according your rdma env
+          - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3
+          - --chunked-prefill-size
+          - "524288"
+          - --max-prefill-tokens
+          - "32768"
+          - --page-size
+          - "64"
+          #          - --init-expert-location
+          #          - /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json
+          - --ep-dispatch-algorithm
+          - dynamic
+          - --eplb-algorithm
+          - deepseek
+          #          - --deepep-config
+          #          -  /home/aiges/tuned/tuned_8sms.json
+          - --enable-dp-lm-head
+          - --enable-dp-attention
+          - --dp-size
+          - "16"
+          - --disable-radix-cache
+          - --moe-a2a-backend
+          - deepep
+          - --disaggregation-mode
+          - prefill
+          - --mem-fraction-static
+          - "0.7"
+          - --context-length
+          - "32768"
+          - --tp
+          - "16"
+          - --dist-init-addr
+          - $(LWS_LEADER_ADDRESS):20102
+          - --nnodes
+          - $(LWS_GROUP_SIZE)
+          - --node-rank
+          - $(LWS_WORKER_INDEX)
+          - --trust-remote-code
+          - --ep-num-redundant-experts
+          - "32"
+          - --moe-dense-tp-size
+          - "1"
+          - --max-running-requests
+          - "1024"
+          env:
+#          - name: NVSHMEM_HCA_PE_MAPPING
+#            value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
+#          - name: NVSHMEM_HCA_LIST
+#            value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1"
+          - name: NVSHMEM_IB_GID_INDEX
+            value: "3"
+          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
+            value: "1"
+          - name: SGLANG_SET_CPU_AFFINITY
+            value: "true"
+          - name: SGLANG_ENABLE_JIT_DEEPGEMM
+            value: "1"
+          - name: NCCL_IB_QPS_PER_CONNECTION
+            value: "8"
+          - name: NCCL_IB_SPLIT_DATA_ON_QPS
+            value: "1"
+          - name: NCCL_NET_PLUGIN
+            value: none
+          - name: NCCL_IB_TC
+            value: "136"
+          - name: NCCL_MIN_NCHANNELS
+            value: "4"
+          - name: MC_TE_METRIC
+            value: "false"
+          - name: NCCL_IB_SL
+            value: "5"
+          - name: NCCL_IB_HCA
+            value: ^=mlx5_0,mlx5_5,mlx5_6
+          - name: LWS_WORKER_INDEX
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+          image: lmsysorg/sglang:deepep
+          name: sglang-leader
+          ports:
+          - containerPort: 30000
+            protocol: TCP
+          readinessProbe:
+            periodSeconds: 30
+            tcpSocket:
+              port: 30000
+          resources:
+            limits:
+              nvidia.com/gpu: "8"
+          securityContext:
+            capabilities:
+              add:
+              - IPC_LOCK
+            privileged: true
+          volumeMounts:
+          - mountPath: /dev/shm
+            name: dshm
+          - mountPath: /work/models
+            name: model
+          - mountPath: /dev/infiniband
+            name: ib
+          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
+            name: cf
+          - mountPath: /root/.cache
+            name: sgl-cache
+        dnsPolicy: ClusterFirstWithHostNet
+        hostIPC: true
+        hostNetwork: true
+        nodeSelector:
+          pd: "yes"
+        tolerations:
+        - key: pd
+          operator: Exists
+        - key: node-role
+          operator: Exists
+        volumes:
+        - emptyDir:
+            medium: Memory
+          name: dshm
+        - hostPath:
+            # modify according to you deployment env
+            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
+          name: model
+        - hostPath:
+            path: /dev/infiniband
+          name: ib
+        - hostPath:
+            # modify according to you deployment env
+            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
+          name: cf
+        - hostPath:
+            # modify according to you deployment env
+            path: /data1/sgl_cache
+            type: DirectoryOrCreate
+          name: sgl-cache
+    restartPolicy: RecreateGroupOnPodRestart
+    size: 2
+    workerTemplate:
+      metadata: {}
+      spec:
+        containers:
+        - command:
+          - python3
+          - -m
+          - sglang.launch_server
+          - --model-path
+          - /work/models
+          - --disaggregation-ib-device
+          - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3
+          - --chunked-prefill-size
+          - "524288"
+          - --max-prefill-tokens
+          - "32768"
+          - --page-size
+          - "64"
+          #- --init-expert-location
+          #- /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json
+          - --ep-dispatch-algorithm
+          - dynamic
+          - --eplb-algorithm
+          - deepseek
+#          - --deepep-config
+#          -  /home/aiges/tuned/tuned_8sms.json
+          - --enable-dp-lm-head
+          - --enable-dp-attention
+          - --dp-size
+          - "16"
+          - --disable-radix-cache
+          - --moe-a2a-backend
+          - deepep
+          - --disaggregation-mode
+          - prefill
+          - --mem-fraction-static
+          - "0.7"
+          - --context-length
+          - "32768"
+          - --tp
+          - "16"
+          - --dist-init-addr
+          - $(LWS_LEADER_ADDRESS):20102
+          - --nnodes
+          - $(LWS_GROUP_SIZE)
+          - --node-rank
+          - $(LWS_WORKER_INDEX)
+          - --trust-remote-code
+          - --ep-num-redundant-experts
+          - "32"
+          - --moe-dense-tp-size
+          - "1"
+          - --max-running-requests
+          - "1024"
+          env:
+          - name: SGLANG_SET_CPU_AFFINITY
+            value: "true"
+          - name: SGLANG_HACK_DEEPEP_NUM_SMS
+            value: "8"
+          - name: SGLANG_HACK_DEEPEP_NEW_MODE
+            value: "0"
+#          - name: NVSHMEM_HCA_PE_MAPPING
+#            value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
+#          - name: NVSHMEM_HCA_LIST
+#            value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1"
+          - name: NCCL_IB_HCA
+            value: ^=mlx5_0,mlx5_5,mlx5_6
+          - name: NVSHMEM_IB_TRAFFIC_CLASS
+            value: "16"
+          - name: NVSHMEM_IB_GID_INDEX
+            value: "3"
+          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
+            value: "1"
+          - name: CUDA_LAUNCH_BLOCKING
+            value: "0"
+          - name: SGLANG_MOONCAKE_TRANS_THREAD
+            value: "8"
+          - name: SGLANG_ENABLE_JIT_DEEPGEMM
+            value: "1"
+          - name: SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD
+            value: "0"
+          - name: NCCL_IB_QPS_PER_CONNECTION
+            value: "8"
+          - name: NCCL_IB_SPLIT_DATA_ON_QPS
+            value: "1"
+          - name: NCCL_NET_PLUGIN
+            value: none
+          - name: NCCL_IB_TC
+            value: "136"
+          - name: NCCL_MIN_NCHANNELS
+            value: "4"
+          - name: MC_TE_METRIC
+            value: "true"
+          - name: NCCL_IB_SL
+            value: "5"
+          - name: LWS_WORKER_INDEX
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+          image: lmsysorg/sglang:deepep
+          name: sglang-worker
+          ports:
+          - containerPort: 30001
+            protocol: TCP
+          resources:
+            limits:
+              nvidia.com/gpu: "8"
+          securityContext:
+            capabilities:
+              add:
+              - IPC_LOCK
+            privileged: true
+          volumeMounts:
+
+          - mountPath: /root/.cache
+            name: sgl-cache
+          - mountPath: /dev/shm
+            name: dshm
+          - mountPath: /work/models
+            name: model
+          - mountPath: /dev/infiniband
+            name: ib
+          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
+            name: cf
+        dnsPolicy: ClusterFirstWithHostNet
+        hostIPC: true
+        hostNetwork: true
+        nodeSelector:
+          pd: "yes"
+        tolerations:
+        - key: pd
+          operator: Exists
+        - key: node-role
+          operator: Exists
+        volumes:
+        - emptyDir:
+            medium: Memory
+          name: dshm
+        - hostPath:
+            path: /dev/infiniband
+          name: ib
+        - hostPath:
+            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
+          name: model
+        - hostPath:
+            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
+          name: cf
+        - hostPath:
+            path: /data1/sgl_cache
+            type: DirectoryOrCreate
+          name: sgl-cache
+
+```
+
+### Decode
+
+Decode node deployment manifest file [decode.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d.yaml)
+
+*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*
+
+```yaml Config
+apiVersion: leaderworkerset.x-k8s.io/v1
+kind: LeaderWorkerSet
+metadata:
+  name: deepseekr10528-decode-main
+spec:
+  leaderWorkerTemplate:
+    leaderTemplate:
+      metadata:
+        labels:
+          role: leader
+      spec:
+        containers:
+        - command:
+          - python3
+          - -m
+          - sglang.launch_server
+          - --port
+          - "30000"
+          - --host
+          - "0.0.0.0"
+          - --model-path
+          - /work/models
+          - --chunked-prefill-size
+          - "262144"
+          - --page-size
+          - "64"
+          - --enable-dp-attention
+          - --enable-dp-lm-head
+          - --dp-size
+          - "16"
+          - --moe-a2a-backend
+          - deepep
+          - --disaggregation-mode
+          - decode
+          - --mem-fraction-static
+          -  "0.849"
+          - --context-length
+          - "32768"
+          - --disaggregation-ib-device
+          - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
+          - --cuda-graph-max-bs-decode
+          - "64"
+          - --max-running-requests
+          - "2048"
+          - --tp-size
+          - "16" # Size of Tensor Parallelism
+          - --dist-init-addr
+          - $(LWS_LEADER_ADDRESS):20102
+          - --nnodes
+          - $(LWS_GROUP_SIZE)
+          - --node-rank
+          - $(LWS_WORKER_INDEX)
+          - --trust-remote-code
+          - --ep-num-redundant-experts
+          - "32"
+          - --moe-dense-tp-size
+          - "1"
+          env:
+          - name: CUDA_LAUNCH_BLOCKING
+            value: "0"
+          - name: NVSHMEM_IB_GID_INDEX
+            value: "3"
+          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
+            value: "1"
+          - name:  NCCL_IB_QPS_PER_CONNECTION
+            value: "8"
+          - name: NCCL_IB_SPLIT_DATA_ON_QPS
+            value: "1"
+          - name: NCCL_NET_PLUGIN
+            value: "none"
+          - name: NCCL_IB_TC
+            value: "136"
+          - name: NCCL_MIN_NCHANNELS
+            value: "4"
+          - name: NCCL_IB_SL
+            value: "5"
+          - name: MC_TE_METRIC
+            value: "true"
+          - name: SGLANG_MOONCAKE_TRANS_THREAD
+            value: "16"
+          - name: SGLANG_ENABLE_JIT_DEEPGEMM
+            value: "1"
+          - name: NCCL_IB_HCA
+            value: ^=mlx5_0,mlx5_5,mlx5_6
+          - name: LWS_WORKER_INDEX
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+          image: lmsysorg/sglang:deepep
+          name: sglang-leader
+          ports:
+          - containerPort: 30000
+            protocol: TCP
+          readinessProbe:
+            periodSeconds: 30
+            tcpSocket:
+              port: 30000
+          resources:
+            limits:
+              nvidia.com/gpu: "8"
+          securityContext:
+            capabilities:
+              add:
+              - IPC_LOCK
+            privileged: true
+          volumeMounts:
+          - mountPath: /root/.cache
+            name: sgl-cache
+          - mountPath: /dev/shm
+            name: dshm
+          - mountPath: /work/models
+            name: model
+          - mountPath: /dev/infiniband
+            name: ib
+          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
+            name: cf
+        dnsPolicy: ClusterFirstWithHostNet
+        hostIPC: true
+        hostNetwork: true
+        nodeSelector:
+          pd: "yes"
+        tolerations:
+        - key: pd
+          operator: Exists
+        - key: node-role
+          operator: Exists
+        volumes:
+        - hostPath:
+            path: /data1/sgl_cache1
+            type: DirectoryOrCreate
+          name: sgl-cache
+        - emptyDir:
+            medium: Memory
+          name: dshm
+        - hostPath:
+            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
+          name: model
+        - hostPath:
+            path: /dev/infiniband
+          name: ib
+        - hostPath:
+            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
+          name: cf
+    restartPolicy: RecreateGroupOnPodRestart
+    size:  2
+    workerTemplate:
+      metadata: {}
+      spec:
+        containers:
+        - command:
+          - python3
+          - -m
+          - sglang.launch_server
+          - --model-path
+          - /work/models
+          - --chunked-prefill-size
+          - "262144"
+          - --page-size
+          - "64"
+          - --enable-dp-attention
+          - --enable-dp-lm-head
+            #- --enable-two-batch-overlap
+          - --dp-size
+          - "16"
+          - --moe-a2a-backend
+          - deepep
+          - --disaggregation-mode
+          - decode
+          - --mem-fraction-static
+          -  "0.849"
+          - --context-length
+          - "32768"
+          - --disaggregation-ib-device
+          # should modify according your rdma env
+          - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
+          - --cuda-graph-max-bs-decode
+          - "64"
+          - --max-running-requests
+          - "2048"
+          - --tp-size
+          - "16" # Size of Tensor Parallelism
+          - --dist-init-addr
+          - $(LWS_LEADER_ADDRESS):20102
+          - --nnodes
+          - $(LWS_GROUP_SIZE)
+          - --node-rank
+          - $(LWS_WORKER_INDEX)
+          - --trust-remote-code
+          - --ep-num-redundant-experts
+          - "32"
+          - --moe-dense-tp-size
+          - "1"
+          env:
+          - name: SGLANG_HACK_DEEPEP_NUM_SMS
+            value: "24"
+          - name: SGLANG_HACK_DEEPEP_NEW_MODE
+            value: "0"
+          - name: NVSHMEM_IB_TRAFFIC_CLASS
+            value: "16"
+          - name: NVSHMEM_IB_GID_INDEX
+            value: "3"
+          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
+            value: "1"
+          - name:  NCCL_IB_QPS_PER_CONNECTION
+            value: "8"
+          - name: NCCL_IB_SPLIT_DATA_ON_QPS
+            value: "1"
+          - name: NCCL_NET_PLUGIN
+            value: "none"
+          - name: NCCL_IB_TC
+            value: "136"
+          - name: NCCL_MIN_NCHANNELS
+            value: "4"
+          - name: MC_TE_METRIC
+            value: "true"
+          - name: NCCL_IB_SL
+            value: "5"
+          - name: SGLANG_MOONCAKE_TRANS_THREAD
+            value: "16"
+          - name: SGLANG_ENABLE_JIT_DEEPGEMM
+            value: "1"
+          - name: NCCL_IB_HCA
+            value: ^=mlx5_0,mlx5_5,mlx5_6
+          - name: LWS_WORKER_INDEX
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+          image: lmsysorg/sglang:deepep
+          name: sglang-worker
+          ports:
+          - containerPort: 30001
+          resources:
+            limits:
+              nvidia.com/gpu: "8"
+          securityContext:
+            capabilities:
+              add:
+              - IPC_LOCK
+            privileged: true
+          volumeMounts:
+          - mountPath: /root/.cache
+            name: sgl-cache
+          - mountPath: /dev/shm
+            name: dshm
+          - mountPath: /work/models
+            name: model
+          - mountPath: /dev/infiniband
+            name: ib
+          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
+            name: cf
+        dnsPolicy: ClusterFirstWithHostNet
+        hostIPC: true
+        hostNetwork: true
+        nodeSelector:
+          pd: "yes"
+        tolerations:
+        - key: pd
+          operator: Exists
+        - key: node-role
+          operator: Exists
+        volumes:
+        - hostPath:
+            path: /data1/sgl_cache1
+            type: DirectoryOrCreate
+          name: sgl-cache
+        - emptyDir:
+            medium: Memory
+          name: dshm
+        - hostPath:
+            path: /dev/infiniband
+          name: ib
+        - hostPath:
+            # modify according to you deployment env
+            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
+          name: model
+        - hostPath:
+            # modify according to you deployment env
+            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
+          name: cf
+  networkConfig:
+    subdomainPolicy: Shared
+  replicas: 1
+  rolloutStrategy:
+    rollingUpdateConfiguration:
+      maxSurge: 0
+      maxUnavailable: 1
+    type: RollingUpdate
+  startupPolicy: LeaderCreated
+```
+
+Execute separately:
+
+```bash Command
+kubectl apply -f p.yaml
+kubectl apply -f d.yaml
+```
+
+At this point, we have completed the deployment of the 1P1D SGLang engine part.
+
+To allow our users to directly experience the model API, we still need a load balancer to handle sequential calls between prefill and decode. Different companies implement LBs differently, and the community will also officially release a new LB component written in Rust in the near future.
+
+Currently, we use a static K8S service + minilb approach to implement model API calls.
+
+### Creating Service for Prefill and Decode
+
+#### Create prefill k8s service
+[p-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p-svc.yaml)
+```yaml Config
+apiVersion: v1
+kind: Service
+metadata:
+  name: deepseekr10528-prefill-main
+spec:
+  selector:
+    leaderworkerset.sigs.k8s.io/name: deepseekr10528-prefill-main
+    role: leader
+  ports:
+    - protocol: TCP
+      port: 30000
+      targetPort: 30000
+```
+Execute `kubectl apply -f p-svc.yaml`
+
+#### Create decode k8s service
+[d-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d-svc.yaml)
+```yaml Config
+apiVersion: v1
+kind: Service
+metadata:
+  name: deepseekr10528-decode-main
+spec:
+  selector:
+    leaderworkerset.sigs.k8s.io/name: deepseekr10528-decode-main
+    role: leader
+  ports:
+    - protocol: TCP
+      port: 30000
+      targetPort: 30000
+```
+Execute `kubectl apply -f d-svc.yaml`
+
+#### Deploy minilb and lb service
+[lb.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/lb.yaml)
+```yaml Config
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: deepseekr10528-lb-main
+  labels:
+    app: deepseekr10528-lb
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: deepseekr10528-lb
+  template:
+    metadata:
+      labels:
+        app: deepseekr10528-lb
+    spec:
+      nodeSelector:
+          pd: "yes"
+      tolerations:
+        - key: pd
+          operator: Exists
+        - key: node-role
+          operator: Exists
+      containers:
+        - name: sgl-minilb
+          image: lmsysorg/sglang:deepep
+          command:
+          - python
+          - -m
+          - sglang_router.launch_router
+          - --pd-disaggregation
+          - --prefill
+          - http://deepseekr10528-prefill-main:30000
+          - --decode
+          - http://deepseekr10528-decode-main:30000
+          - --host
+          - 0.0.0.0
+          - --port
+          -  "8000"
+          ports:
+            - containerPort: 8000
+***
+apiVersion: v1
+kind: Service
+metadata:
+  name: deepseekr10528-lb-service
+spec:
+  type: NodePort
+  selector:
+    app: deepseekr10528-lb
+  ports:
+    - protocol: TCP
+      port: 8000         # Service Port（In-Cluster）
+      targetPort: 8000   # Exposed Container
+      nodePort: 30800
+```
+Execute `kubectl apply -f lb.yaml`
+
+After waiting for all model deployments to succeed, you will get the following output:
+
+```bash Command
+[root@ecs-001]# kubectl get po
+deepseekr10528-decode-main-0             1/1     Running   0          74m
+deepseekr10528-decode-main-0-1           1/1     Running   0          74m
+deepseekr10528-lb-main-9c5dbfc57-6lcbd   1/1     Running   0          22m
+deepseekr10528-prefill-main-0            1/1     Running   0          74m
+deepseekr10528-prefill-main-0-1          1/1     Running   0          74m
+[root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl  get svc |grep dee
+deepseekr10528-decode-main    ClusterIP   None             <none>        <none>           97m
+deepseekr10528-lb-service     NodePort    172.16.242.169   <none>        8000:30800/TCP   22m
+deepseekr10528-prefill-main   ClusterIP   None             <none>        <none>           97m
+```
+
+At this point, select a nodePort:30800 to access:
+
+```bash Command
+[root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \
+>     -H "Content-Type: application/json" \
+>     -H "Authorization: Bearer None" \
+>     -d '{
+>        "rid":"ccccdd",
+>         "model": "r1",
+>         "messages": [
+>             {"role": "system", "content": "0: You are a helpful AI assistant"},
+>             {"role": "user", "content": "你是谁？."}
+>         ],
+>         "max_tokens":221
+>     }'
+{"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n</think>\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-R1**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}}
+
+```
+## FAQ
+
+1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments.
+
+2. Some preset, optimized configurations for EPLB are not used here. You can adjust them according to [6017](https://github.com/sgl-project/sglang/issues/6017) as needed.

docs_new/docs/references/multi_node_deployment/multi_node.mdx

@@ -0,0 +1,103 @@
+---
+title: "Multi-Node Deployment"
+metatags:
+    description: "SGLang multi-node: Llama 405B on 2 nodes, DeepSeek V3/R1, SLURM cluster deployment examples."
+---
+## Llama 3.1 405B
+
+**Run 405B (fp16) on Two Nodes**
+
+```bash Command
+# replace 172.16.4.52:20000 with your own node ip address and port of the first node
+
+python3 -m sglang.launch_server \
+  --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \
+  --tp 16 \
+  --dist-init-addr 172.16.4.52:20000 \
+  --nnodes 2 \
+  --node-rank 0
+
+python3 -m sglang.launch_server \
+  --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \
+  --tp 16 \
+  --dist-init-addr 172.16.4.52:20000 \
+  --nnodes 2 \
+  --node-rank 1
+```
+
+Note that LLama 405B (fp8) can also be launched on a single node.
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8
+```
+
+## DeepSeek V3/R1
+
+Please refer to [DeepSeek documents for reference](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-5-multi-node-deployment).
+
+## Multi-Node Inference on SLURM
+
+This example showcases how to serve SGLang server across multiple nodes by SLURM. Submit the following job to the SLURM cluster.
+
+```bash Command
+#!/bin/bash -l
+
+#SBATCH -o SLURM_Logs/%x_%j_master.out
+#SBATCH -e SLURM_Logs/%x_%j_master.err
+#SBATCH -D ./
+#SBATCH -J Llama-405B-Online-Inference-TP16-SGL
+
+#SBATCH --nodes=2
+#SBATCH --ntasks=2
+#SBATCH --ntasks-per-node=1  # Ensure 1 task per node
+#SBATCH --cpus-per-task=18
+#SBATCH --mem=224GB
+#SBATCH --partition="lmsys.org"
+#SBATCH --gres=gpu:8
+#SBATCH --time=12:00:00
+
+echo "[INFO] Activating environment on node $SLURM_PROCID"
+if ! source ENV_FOLDER/bin/activate; then
+    echo "[ERROR] Failed to activate environment" >&2
+    exit 1
+fi
+
+# Define parameters
+model=MODEL_PATH
+tp_size=16
+
+echo "[INFO] Running inference"
+echo "[INFO] Model: $model"
+echo "[INFO] TP Size: $tp_size"
+
+# Set NCCL initialization address using the hostname of the head node
+HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1)
+NCCL_INIT_ADDR="${HEAD_NODE}:8000"
+echo "[INFO] NCCL_INIT_ADDR: $NCCL_INIT_ADDR"
+
+# Launch the model server on each node using SLURM
+srun --ntasks=2 --nodes=2 --output="SLURM_Logs/%x_%j_node$SLURM_NODEID.out" \
+    --error="SLURM_Logs/%x_%j_node$SLURM_NODEID.err" \
+    python3 -m sglang.launch_server \
+    --model-path "$model" \
+    --grammar-backend "xgrammar" \
+    --tp "$tp_size" \
+    --dist-init-addr "$NCCL_INIT_ADDR" \
+    --nnodes 2 \
+    --node-rank "$SLURM_NODEID" &
+
+# Wait for the NCCL server to be ready on port 30000
+while ! nc -z "$HEAD_NODE" 30000; do
+    sleep 1
+    echo "[INFO] Waiting for $HEAD_NODE:30000 to accept connections"
+done
+
+echo "[INFO] $HEAD_NODE:30000 is ready to accept connections"
+
+# Keep the script running until the SLURM job times out
+wait
+```
+
+Then, you can test the server by sending requests following other [documents](../../basic_usage/openai_api_completions).
+
+Thanks for [aflah02](https://github.com/aflah02) for providing the example, based on his [blog post](https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang).

docs_new/docs/references/multi_node_deployment/multi_node_index.mdx

@@ -0,0 +1,11 @@
+---
+title: "Multi-Node Deployment"
+metatags:
+    description: "SGLang multi-node deployment index: K8s, LWS, RBG, PD disaggregation guides."
+---
+- [Multi Node](./multi_node)
+- [Deploy On K8S](./deploy_on_k8s)
+- [Lws Pd Deploy](./lws_pd/lws_pd_deploy)
+- [Deepseekv32 Pd](./rbg_pd/deepseekv32_pd)
+- [Deploying DeepSeek with PD Disaggregation on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
+- [Deploying Kimi K2 with PD Disaggregation on 128 H200 GPUs](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/)

docs_new/docs/references/multi_node_deployment/multi_node_index.rst

@@ -0,0 +1,14 @@
+Multi-Node Deployment
+=====================
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Multi-Node Deployment
+
+   multi_node.md
+   deploy_on_k8s.md
+   lws_pd/lws_pd_deploy.md
+   rbg_pd/deepseekv32_pd.md
+
+- `Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs <https://lmsys.org/blog/2025-05-05-large-scale-ep/>`_
+- `Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs <https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/>`_

docs_new/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd.mdx

@@ -0,0 +1,570 @@
+---
+title: "DeepSeekV32-Exp RBG Based PD Deploy"
+metatags:
+    description: "SGLang DeepSeek V3.2 RBG deployment: RoleBasedGroup PD disaggregation on Kubernetes."
+---
+## 0. Prerequisites
+
+1. k8s >=1.26
+2. lws installed on k8s.
+3. rbg installed on k8s.
+
+For RBG installation, please refer to: https://github.com/sgl-project/rbg
+
+## 1. Image Preparation
+
+`lmsysorg/sglang:latest`
+
+
+### 2. All In One manifest file
+
+*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*
+
+rbg-dsv32.yml
+
+```yaml Config
+apiVersion: workloads.x-k8s.io/v1alpha1
+kind: RoleBasedGroup
+metadata:
+  name: deepseek-rbg-32exp
+  namespace: default
+spec:
+  roles:
+    - name: prefill
+      replicas: 1
+      workload:
+        apiVersion: leaderworkerset.x-k8s.io/v1
+        kind: LeaderWorkerSet
+      restartPolicy: None
+      leaderWorkerSet:
+        size: 1
+        patchLeaderTemplate:
+          metadata:
+            labels:
+              role: leader
+              pd_role: prefill
+          spec:
+            containers:
+            - command:
+              - python3
+              - -m
+              - sglang.launch_server
+              - --model-path
+              - /work/models
+              - --port
+              - "30000"
+              - --trust-remote
+              - --host
+              -  0.0.0.0
+              - --disaggregation-ib-device
+              -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
+              - --disable-radix-cache
+              - --chunked-prefill-size
+              - "131072"
+              - --page-size
+              - "64"
+    #          - --enable-eplb
+              - --ep-dispatch-algorithm
+              - dynamic
+              - --eplb-algorithm
+              - deepseek
+              - --enable-dp-lm-head
+              - --enable-dp-attention
+              - --dp-size
+              - "8"
+              - --moe-a2a-backend
+              - deepep
+              - --deepep-mode
+              - normal
+              - --disaggregation-mode
+              - prefill
+              - --mem-fraction-static
+              - "0.8"
+              - --max-prefill-tokens
+              - "32768"
+              - --context-length
+              - "32768"
+              - --tp
+              - "8"
+              - --dist-init-addr
+              - $(LWS_LEADER_ADDRESS):20102
+              - --nnodes
+              - $(LWS_GROUP_SIZE)
+              - --node-rank
+              - $(LWS_WORKER_INDEX)
+              - --trust-remote-code
+              - --ep-num-redundant-experts
+              - "32"
+              - --moe-dense-tp-size
+              - "1"
+              - --max-running-requests
+              - "1024"
+              env:
+              - name: LWS_WORKER_INDEX
+                valueFrom:
+                  fieldRef:
+                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+              livenessProbe:
+                failureThreshold: 3000
+                httpGet:
+                  path: /health
+                  port: 30000
+                initialDelaySeconds: 300
+                periodSeconds: 60
+                successThreshold: 1
+                timeoutSeconds: 10
+              readinessProbe:
+                failureThreshold: 20
+                httpGet:
+                  path: /health
+                  port: 30000
+                periodSeconds: 30
+                successThreshold: 1
+                timeoutSeconds: 10
+              name: sglang
+              ports:
+              - containerPort: 30000
+                name: sglang-http
+                protocol: TCP
+
+        patchWorkerTemplate: {}
+      template:
+        metadata:
+          labels:
+            inference-framework: sglang
+            inference-stack.io/monitoring: "enabled"
+        spec:
+            containers:
+            - name: sglang
+              image: lmsysorg/sglang:latest
+              env:
+                - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
+                  value: "1"
+                - name: CUDA_LAUNCH_BLOCKING
+                  value: "0"
+                - name:  SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
+                  value: "1000000000"
+                - name: NVSHMEM_IB_TRAFFIC_CLASS
+                  value: "16"
+                - name: NVSHMEM_DISABLE_P2P
+                  value: "0"
+                - name: ENABLE_METRICS
+                  value: "true"
+                - name: NVSHMEM_IB_GID_INDEX
+                  value: "3"
+                - name: NVSHMEM_IB_SL
+                  value: "5"
+                - name: SGLANG_SET_CPU_AFFINITY
+                  value: "true"
+                - name: SGL_ENABLE_JIT_DEEPGEMM
+                  value: "1"
+                - name:  NCCL_IB_QPS_PER_CONNECTION
+                  value: "8"
+                - name: NCCL_IB_SPLIT_DATA_ON_QPS
+                  value: "1"
+                - name: NCCL_NET_PLUGIN
+                  value: "none"
+                - name: NCCL_IB_TC
+                  value: "136"
+                - name: NCCL_IB_SL
+                  value: "5"
+                - name: NCCL_IB_TIMEOUT
+                  value: "22"
+                - name: NCCL_IB_GID_INDEX
+                  value: "3"
+                - name: NCCL_MIN_NCHANNELS
+                  value: "4"
+                - name: NCCL_SOCKET_IFNAME
+                  value: bond0
+                - name: GLOO_SOCKET_IFNAME
+                  value: bond0
+                - name: NCCL_IB_HCA
+                  value: ^=mlx5_0,mlx5_5,mlx5_6
+                - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
+                  value: "bond0"
+                - name: MC_TE_METRIC
+                  value: "false"
+              resources:
+                limits:
+                  nvidia.com/gpu: "8"
+              securityContext:
+                capabilities:
+                  add:
+                  - IPC_LOCK
+                privileged: true
+              volumeMounts:
+                - mountPath: /root/.cache
+                  name: sgl-cache
+                - mountPath: /dev/shm
+                  name: dshm
+                - mountPath: /work/models
+                  name: model
+                - mountPath: /dev/infiniband
+                  name: ib
+                - mountPath: /sgl-workspace/sglang
+                  name: src
+
+            dnsPolicy: ClusterFirstWithHostNet
+            hostIPC: true
+            hostNetwork: true
+            nodeSelector:
+              pd: "yes"
+            tolerations:
+              - key: pd
+                operator: Exists
+            volumes:
+            - hostPath:
+                path: /var/run/sys-topology
+              name: topo
+            - hostPath:
+                path: /data1/sgl_cache4
+                type: DirectoryOrCreate
+              name: sgl-cache
+            - emptyDir:
+                medium: Memory
+              name: dshm
+            - hostPath:
+                path: /data/DeepSeek-V3.2-Exp
+              name: model
+            - hostPath:
+                path: /dev/infiniband
+              name: ib
+            - hostPath:
+                path: /data/src/sglang
+                type: DirectoryOrCreate
+              name: src
+
+    - name: decode
+      replicas: 1
+      workload:
+        apiVersion: leaderworkerset.x-k8s.io/v1
+        kind: LeaderWorkerSet
+      leaderWorkerSet:
+        size: 1
+        patchLeaderTemplate:
+          metadata:
+            labels:
+              role: leader
+              pd_role: decode
+          spec:
+            containers:
+            - command:
+                  - python3
+                  - -m
+                  - sglang.launch_server
+                  - --model-path
+                  - /work/models
+                  - --port
+                  - "30000"
+                  - --trust-remote
+                  - --host
+                  -  0.0.0.0
+                  - --disaggregation-ib-device
+                  -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
+                  - --chunked-prefill-size
+                  - "131072"
+                  - --eplb-rebalance-layers-per-chunk
+                  - "29"
+                  - --page-size
+                  - "64"
+                  - --enable-dp-attention
+                  - --enable-dp-lm-head
+                  - --dp-size
+                  - "8"
+                  - --moe-a2a-backend
+                  - deepep
+                  - --deepep-mode
+                  - low_latency
+                  - --disaggregation-mode
+                  - decode
+                  - --mem-fraction-static
+                  -  "0.8"
+                  - --context-length
+                  - "32768"
+                  - --max-running-requests
+                  - "2048"
+                  - --tp-size
+                  - "8" # Size of Tensor Parallelism
+                  - --cuda-graph-max-bs-decode
+                  - "16"
+                  - --dist-init-addr
+                  - $(LWS_LEADER_ADDRESS):20102
+                  - --nnodes
+                  - $(LWS_GROUP_SIZE)
+                  - --node-rank
+                  - $(LWS_WORKER_INDEX)
+                  - --trust-remote-code
+                  - --ep-num-redundant-experts
+                  - "32"
+                  - --moe-dense-tp-size
+                  - "1"
+              env:
+              - name: LWS_WORKER_INDEX
+                valueFrom:
+                  fieldRef:
+                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+              livenessProbe:
+                failureThreshold: 30000
+                httpGet:
+                  path: /health
+                  port: 30000
+                initialDelaySeconds: 300
+                periodSeconds: 60
+                successThreshold: 1
+                timeoutSeconds: 10
+              name: sglang
+              readinessProbe:
+                failureThreshold: 20
+                httpGet:
+                  path: /health
+                  port: 30000
+                periodSeconds: 30
+                successThreshold: 1
+                timeoutSeconds: 10
+        patchWorkerTemplate:
+          spec:
+            containers:
+            - command:
+                - python3
+                - -m
+                - sglang.launch_server
+                - --model-path
+                - /work/models
+                - --crash-dump-folder
+                -  /log
+                - --chunked-prefill-size
+                - "262144"
+                - --eplb-rebalance-layers-per-chunk
+                - "29"
+                - --page-size
+                - "64"
+                - --enable-dp-attention
+                - --enable-dp-lm-head
+                - --dp-size
+                - "32"
+                - --moe-a2a-backend
+                - "deepep"
+                - --deepep-mode
+                - low_latency
+                - --disaggregation-mode
+                - decode
+                - --mem-fraction-static
+                -  "0.849"
+                - --context-length
+                - "32768"
+                - --disaggregation-ib-device
+                -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
+                - --max-running-requests
+                - "4096"
+                - --cuda-graph-max-bs-decode
+                - "16"
+                - --tp-size
+                - "8" # Size of Tensor Parallelism
+                - --dist-init-addr
+                - $(LWS_LEADER_ADDRESS):20102
+                - --nnodes
+                - $(LWS_GROUP_SIZE)
+                - --node-rank
+                - $(LWS_WORKER_INDEX)
+                - --trust-remote-code
+                - --ep-num-redundant-experts
+                - "32"
+                - --moe-dense-tp-size
+                - "1"
+              env:
+              - name: LWS_WORKER_INDEX
+                valueFrom:
+                  fieldRef:
+                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
+              name: sglang
+      template:
+        metadata:
+          labels:
+            inference-framework: sglang-unuse
+            inference-stack.io/monitoring: "enabled"
+        spec:
+            containers:
+            - image: lmsysorg/sglang:latest
+              name: sglang
+              resources:
+                limits:
+                  nvidia.com/gpu: "8"
+              securityContext:
+                capabilities:
+                  add:
+                  - IPC_LOCK
+                privileged: true
+              volumeMounts:
+                - mountPath: /root/.cache
+                  name: sgl-cache
+                - mountPath: /dev/shm
+                  name: dshm
+                - mountPath: /work/models
+                  name: model
+                - mountPath: /dev/infiniband
+                  name: ib
+                - mountPath: /sgl-workspace/sglang
+                  name: src
+              env:
+                - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
+                  value: "1"
+                - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
+                  value: "100000000"
+                - name: NVSHMEM_DISABLE_P2P
+                  value: "0"
+                - name: NVSHMEM_IB_TRAFFIC_CLASS
+                  value: "16"
+                - name: NVSHMEM_IB_SL
+                  value: "5"
+                - name: ENABLE_METRICS
+                  value: "true"
+                - name: CUDA_LAUNCH_BLOCKING
+                  value: "0"
+                - name: NVSHMEM_IB_GID_INDEX
+                  value: "3"
+                - name:  NCCL_IB_QPS_PER_CONNECTION
+                  value: "8"
+                - name: NCCL_IB_SPLIT_DATA_ON_QPS
+                  value: "1"
+                - name: NCCL_NET_PLUGIN
+                  value: "none"
+                - name: NCCL_IB_TC
+                  value: "136"
+                - name: NCCL_IB_SL
+                  value: "5"
+                - name: NCCL_IB_TIMEOUT
+                  value: "22"
+                - name: NCCL_IB_GID_INDEX
+                  value: "3"
+                - name: NCCL_MIN_NCHANNELS
+                  value: "4"
+                - name: NCCL_SOCKET_IFNAME
+                  value: bond0
+                - name: GLOO_SOCKET_IFNAME
+                  value: bond0
+                - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
+                  value: "bond0"
+                - name: NCCL_IB_HCA
+                  value: ^=mlx5_0,mlx5_5,mlx5_6
+                - name: MC_TE_METRIC
+                  value: "false"
+                - name: SGL_ENABLE_JIT_DEEPGEMM
+                  value: "1"
+            dnsPolicy: ClusterFirstWithHostNet
+            hostIPC: true
+            hostNetwork: true
+            nodeSelector:
+              pd: "yes"
+            tolerations:
+            - key: pd
+              operator: Exists
+            volumes:
+            - hostPath:
+                path: /var/run/sys-topology
+              name: topo
+            - hostPath:
+                path: /data1/sgl_cache4
+                type: DirectoryOrCreate
+              name: sgl-cache
+            - hostPath:
+                path: /data/src/sglang
+                type: DirectoryOrCreate
+              name: src
+            - emptyDir:
+                medium: Memory
+              name: dshm
+            - hostPath:
+                path: /data/DeepSeek-V3.2-Exp
+              name: model
+            - hostPath:
+                path: /dev/infiniband
+              name: ib
+    - name: router
+      replicas: 1
+      dependencies: [ "decode", "prefill" ]
+      template:
+        spec:
+          containers:
+            - name: scheduler
+              image: lmsysorg/sglang:latest
+              command:
+              - sh
+              - -c
+              - >
+                python3 -m sglang_router.launch_router
+                --host 0.0.0.0
+                --port 8080
+                --pd-disaggregation
+                --policy random
+                --service-discovery
+                --service-discovery-namespace ${NAMESPACE}
+                --service-discovery-port 30000
+                --prefill-selector pd_role=prefill
+                --decode-selector pd_role=decode
+                --max-payload-size 2147483648
+                --worker-startup-timeout-secs 1200
+              env:
+              - name: NAMESPACE
+                valueFrom:
+                  fieldRef:
+                    apiVersion: v1
+                    fieldPath: metadata.namespace
+***
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: deepseek-rbg-32exp
+  name: deepseek-rbg-32exp
+  namespace: default
+spec:
+  ports:
+    - name: http
+      port: 8080
+      protocol: TCP
+      targetPort: 8080
+      nodePort: 30080
+
+  selector:
+    rolebasedgroup.workloads.x-k8s.io/name: deepseek-rbg-32exp
+    rolebasedgroup.workloads.x-k8s.io/role: router
+  type: NodePort
+
+```
+
+```bash Command
+[root@ecs-001]# kubectl get po -n default
+deepseek-rbg-32exp-decode-main-0             1/1     Running   0          74m
+deepseek-rbg-32exp-decode-0-1                1/1     Running   0          74m
+deepseek-rbg-32exp-router-9c5dbfc57          1/1     Running   0          22m
+deepseek-rbg-32exp-prefill-0                 1/1     Running   0          74m
+
+[root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl  get svc |grep dee
+deepseek-rbg-32exp-decode             ClusterIP   None             <none>        <none>           97m
+deepseek-rbg-32exp-router-service     NodePort    172.16.242.169   <none>        8000:30800/TCP   22m
+deepseek-rbg-32exp-prefill            ClusterIP   None             <none>        <none>           97m
+```
+
+At this point, select a nodePort:30800 to access:
+
+```bash Command
+[root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \
+>     -H "Content-Type: application/json" \
+>     -H "Authorization: Bearer None" \
+>     -d '{
+>        "rid":"ccccdd",
+>         "model": "dsv32",
+>         "messages": [
+>             {"role": "system", "content": "0: You are a helpful AI assistant"},
+>             {"role": "user", "content": "你是谁？."}
+>         ],
+>         "max_tokens":221
+>     }'
+{"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"&lt;think&gt;\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n&lt;/think&gt;\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-V32**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}}
+
+```
+## FAQ
+
+1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments.
+
+2. Please ensure that the sglang code in the image has incorporated the changes from [PR #10912](https://github.com/sgl-project/sglang/pull/10912).

docs_new/docs/references/nightly_precision_regression.mdx

@@ -0,0 +1,372 @@
+# Nightly Precision Regression Testing
+
+## Overview
+
+The nightly precision regression framework detects silent numerical regressions in the SGLang serving engine by comparing **per-layer hidden states** between consecutive runs. It runs as a nightly CI job on 8×H200 GPUs and can also be invoked locally for development and debugging.
+
+The framework operates on a **rolling-baseline** model:
+
+1. **Baseline creation or comparison:** Launch the server, send a fixed prompt, dump per-layer hidden states to disk. If a previous baseline exists, compare the new tensors against it using the SGLang tensor comparator. If the comparison passes, the new tensors become the updated baseline.
+2. On the first run (or when the capture shape changes), the dumped tensors are saved as a new baseline with no comparison.
+
+Baselines are stored locally on disk and synced to a **HuggingFace dataset** so they survive across CI runners and can be shared across machines. The HF dataset store is **required** — the test errors if `SGLANG_PRECISION_HF_REPO` is unset.
+
+---
+
+## How It Works
+
+### Step-by-step flow
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│  1. Resolve model config (layer count, capture layers)        │
+│     ↓                                                         │
+│  2. Compute capture_signature (schema, layers, TP, filter)    │
+│     ↓                                                         │
+│  3. Fetch baseline from HF dataset (signature-matched)        │
+│     ↓                                                         │
+│  4. Launch SGLang server with DUMPER enabled                   │
+│     ↓                                                         │
+│  5. POST /dumper/configure  (set layer filter + cleanup)      │
+│     ↓                                                         │
+│  6. POST /v1/chat/completions  (fixed prompt, 2 tokens,       │
+│     ignore_eos=true to force decode path)                      │
+│     ↓                                                         │
+│  7. Kill server; assert decode tensors were captured           │
+│     ↓                                                         │
+│  8. Baseline exists (with matching signature)?                 │
+│     ├── YES → Run comparator → pass/fail                      │
+│     │         ├── PASS  → update baseline, push to HF         │
+│     │         └── FAIL  → push diagnostics to HF              │
+│     └── NO  → copy today's tensors as initial baseline        │
+│                → push to HF as "baseline_established"          │
+│     ↓                                                         │
+│  9. Report summary (stdout + GitHub Step Summary)             │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Key components
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| Test entry point | `test/registered/debug_utils/test_nightly_precision_regression.py` | Orchestrates server launch, dump, compare, and reporting |
+| HF baseline store | `python/sglang/test/precision_baseline_store.py` | Push / fetch / prune baselines on a HuggingFace dataset |
+| Tensor comparator | `python/sglang/srt/debug_utils/comparator/` | Compares two directories of `.pt` tensors, emits JSONL report |
+| Dumper infrastructure | `python/sglang/srt/debug_utils/dumper.py` | Captures per-layer hidden states at runtime |
+| CI workflow | `.github/workflows/nightly-test-nvidia.yml` | Schedules the nightly job on 8×H200 |
+
+---
+
+## What Gets Dumped and Compared
+
+### Strided layer capture
+
+Not every layer is dumped — the framework uses a **strided capture** to reduce I/O and storage overhead. By default, it captures:
+
+- Layer 0 (always)
+- The last layer (always)
+- Every 8th layer in between (configurable via `LAYER_CAPTURE_STRIDE`)
+
+The layer count is resolved automatically from the model's HuggingFace `config.json` (`num_hidden_layers` or `num_layers`). If resolution fails, all layers are captured as a safe fallback.
+
+The dumper filter is built dynamically as a regex matching only the selected layer indices, e.g.:
+
+```
+match(r'^non_intrusive__model\.layers\.(0|7|15|23)\.inputs\.1$', name)
+```
+
+### Decode-path verification
+
+The test generates **2 tokens** with `ignore_eos=True` to ensure the model's decode path is exercised. After the dump, `_assert_decode_captured()` verifies that tensors from the decode step were actually captured (not just prefill). If only prefill tensors are found, the test fails immediately — this catches misconfigurations where `--max-total-tokens` is too low for the decode loop to run.
+
+### Comparator
+
+The comparator computes **relative differences** (`rel_diff`) for each tensor and checks them against a configurable threshold (default `1e-3`). For tensor-parallel models, the `--override-dims` flag tells the comparator how to reduce across TP ranks before comparing:
+
+```
+--override-dims ^non_intrusive__model\.layers\.\d+\.inputs\.1$:bs h[tp:partial]
+```
+
+This sums partial TP contributions along the hidden dimension before computing the diff, so the comparison is semantically correct even with TP > 1.
+
+If the comparator returns exit code 0 but compared **zero layers** (baseline/target name mismatch), the test fails with a diagnostic message rather than silently passing.
+
+### Capture signature
+
+A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP size, and dumper filter) is computed per run. The HF store uses this signature during fetch to ensure only baselines with an identical capture shape are considered. If the signature changes (e.g. you add layers to the capture set or change TP), the framework establishes a fresh baseline instead of erroring on incompatible tensors.
+
+---
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SGLANG_PRECISION_MODELS` | `zai-org/GLM-5.1-FP8` | Comma-separated HuggingFace model IDs to test |
+| `SGLANG_PRECISION_BASELINE_DIR` | `/tmp/sglang_precision_baselines` | Local directory for baseline tensors |
+| `SGLANG_PRECISION_DIFF_THRESHOLD` | `1e-3` | Per-tensor relative diff threshold |
+| `SGLANG_PRECISION_FORCE_UPDATE` | `0` | Set to `1` to skip comparison and unconditionally refresh baseline |
+| `SGLANG_PRECISION_COMMIT` | _(auto-detected from git)_ | Override the sglang commit SHA tagged on push |
+| `SGLANG_PRECISION_HF_REPO` | _(required)_ | HuggingFace dataset repo for cross-runner baseline storage |
+| `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset |
+| `HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset |
+
+---
+
+## CI Integration
+
+### Workflow job
+
+The nightly job `nightly-test-precision-8-gpu-h200` is defined in `.github/workflows/nightly-test-nvidia.yml` and runs on an 8-GPU H200 runner. It is included in the nightly suite via `test/run_suite.py`.
+
+Key CI configuration:
+
+```yaml
+- name: Run precision regression test
+  timeout-minutes: 120
+  env:
+    SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines
+    SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }}
+    SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}
+    HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
+    SGLANG_PRECISION_COMMIT: ${{ github.sha }}
+  run: |
+    cd test
+    python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600
+```
+
+### Required GitHub secrets/variables
+
+| Name | Type | Purpose |
+|------|------|---------|
+| `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset |
+| `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) |
+| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset |
+
+### GitHub Step Summary
+
+When running in CI, the test writes a Markdown table to the GitHub Actions job summary showing each model's status (`PASSED`, `FAILED`, `BASELINE_ESTABLISHED`, or `ERROR`).
+
+---
+
+## HF Dataset Storage Layout
+
+Baselines are organized in the HF dataset as:
+
+```
+<model_sanitized>/<YYYY>/<MM>/<DD>/run-<sha7>/
+├── meta.json                    # Run metadata (model, commit, hardware, thresholds, stats)
+├── comparator_report.jsonl      # Per-tensor comparison results
+└── tensors/
+    ├── layer_0_inputs_1.pt
+    ├── layer_7_inputs_1.pt
+    └── ...
+```
+
+A top-level `manifest.jsonl` tracks all runs with one JSON object per line. Each manifest row carries a `capture_signature` field so that fetch selects only baselines with a matching capture shape.
+
+The `prune_old_runs()` function (callable manually) retains daily runs for 30 days and keeps one run per week beyond that window.
+
+---
+
+## How to Add a New Model
+
+### Option A: Add to the default model list (CI)
+
+Edit the default in `test/registered/debug_utils/test_nightly_precision_regression.py`:
+
+```python
+DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8,your-org/your-model"
+```
+
+Or set the `SGLANG_PRECISION_MODELS` environment variable in the CI workflow to override the default.
+
+### Option B: Run locally for a specific model
+
+```bash
+export SGLANG_PRECISION_MODELS="your-org/your-model"
+export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines"
+export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3"
+export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
+export HF_TOKEN="hf_..."
+
+cd test
+python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
+```
+
+### Step-by-step: adding a model to the nightly CI
+
+1. **Verify the model works with the dumper.** Run locally first to ensure hidden states are captured correctly:
+
+   ```bash
+   export SGLANG_PRECISION_MODELS="your-org/your-model"
+   export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines"
+   export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
+   export HF_TOKEN="hf_..."
+   export SGLANG_PRECISION_FORCE_UPDATE="1"  # first run: establish baseline
+
+   cd test
+   python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
+   ```
+
+2. **Run a comparison pass** (remove `FORCE_UPDATE`):
+
+   ```bash
+   unset SGLANG_PRECISION_FORCE_UPDATE
+   python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
+   ```
+
+   This should report `PASSED` if the engine is numerically stable for the model.
+
+3. **Set the tensor-parallelism size.** If the model requires TP > 1, the test harness defaults to `tp_size=8` for all models. To customize, modify the `ModelLaunchSettings` construction in the test or pass extra server arguments:
+
+   ```python
+   # In setUpClass or via env-driven logic
+   cls.models = [ModelLaunchSettings("your-org/your-model", tp_size=4)]
+   ```
+
+4. **Adjust the diff threshold if needed.** FP8 or quantized models may exhibit larger numerical differences. Set `SGLANG_PRECISION_DIFF_THRESHOLD` to an appropriate value (e.g., `1e-2` for FP8).
+
+5. **Add to the default model list** or configure `SGLANG_PRECISION_MODELS` in the CI workflow.
+
+### Considerations for model-specific adjustments
+
+| Concern | How to handle |
+|---------|--------------|
+| TP size != 8 | Override `tp_size` in `ModelLaunchSettings` or add model-specific logic |
+| Quantized models (FP8, GPTQ) | Loosen `SGLANG_PRECISION_DIFF_THRESHOLD` (e.g., `1e-2`) |
+| Model needs extra server args | Pass them via `ModelLaunchSettings(model, extra_args=["--quantization", "fp8"])` |
+| Model needs different prompt | Modify `PROMPT` constant or make it model-configurable |
+| MoE models with TP partial sums | Already handled by `--override-dims` (`bs h[tp:partial]`) |
+| Fewer/more capture layers | Adjust `LAYER_CAPTURE_STRIDE` (default 8); set lower for smaller models |
+| Decode not captured | Ensure `--max-total-tokens` is well above the scheduler's decode reservation (default 512); the test uses 4096 |
+
+---
+
+## Running Locally
+
+### Prerequisites
+
+- SGLang installed in development mode
+- GPUs matching the model's requirements
+- `huggingface_hub` installed
+- A **HuggingFace dataset** for baseline storage and a write-capable `HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback.
+
+### Quick local test
+
+```bash
+# All three are required — the test errors if SGLANG_PRECISION_HF_REPO is unset.
+export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct"
+export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines"
+export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
+export HF_TOKEN="hf_..."
+
+# First run: establish baseline
+cd test
+python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
+
+# Second run: compare against baseline
+python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
+```
+
+### Force-refresh a baseline
+
+```bash
+export SGLANG_PRECISION_FORCE_UPDATE="1"
+python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
+```
+
+---
+
+## Interpreting Results
+
+### Status codes
+
+| Status | Meaning |
+|--------|---------|
+| `BASELINE_ESTABLISHED` | No prior baseline with a matching signature existed; today's tensors saved as the new baseline |
+| `PASSED` | All per-layer hidden states are within the diff threshold; baseline updated |
+| `FAILED` | One or more layers exceeded the diff threshold, or 0 layers were compared (baseline/target mismatch); diagnostic data pushed to HF |
+| `ERROR` | Server launch, inference, or comparison encountered an unexpected error |
+
+### Output example
+
+```
+============================================================
+Nightly Precision Regression Summary
+============================================================
+Model                                          Status                   Details
+------------------------------------------------------------
+zai-org/GLM-5.1-FP8                           PASSED                   comparison ok, baseline updated
+Qwen/Qwen2.5-0.5B-Instruct                    FAILED                   tensor=layer_23.inputs_1 rel_diff=0.0152
+============================================================
+```
+
+### When a failure is detected
+
+1. The comparator output is saved to `/tmp/nightly_precision_<model>_*.log`
+2. The failing tensors and comparator report are pushed to the HF dataset with `pass_label="failed"` for offline diagnosis
+3. The GitHub Step Summary includes the failure details
+4. The CI job exits with a non-zero status
+
+---
+
+## Baseline Management
+
+### Local baselines
+
+Baselines are stored at:
+```
+$SGLANG_PRECISION_BASELINE_DIR/<model_sanitized>/nightly_precision/*.pt
+```
+
+A `baseline_meta.json` next to the tensors records the timestamp and commit that produced the baseline.
+
+### HF dataset baselines
+
+- **Fetch:** At test start, if no local baseline exists, the latest signature-matched baseline is downloaded from the HF dataset.
+- **Push:** After each run, tensors and metadata are uploaded to the dataset.
+- **Prune:** Use `prune_old_runs()` to garbage-collect old baselines (keeps 30 days of daily runs, one per week after that).
+
+### Refreshing a stale baseline
+
+If an intentional numerical change (e.g., kernel optimization, model refactor) causes a comparison failure:
+
+1. Verify the change is intentional
+2. Set `SGLANG_PRECISION_FORCE_UPDATE=1` and run the test once to establish a new baseline
+3. Commit any necessary threshold adjustments
+
+If you change the capture configuration (stride, TP size, etc.), the `capture_signature` will differ and the framework automatically establishes a fresh baseline — no manual intervention needed.
+
+---
+
+## Known Limitations
+
+### Baseline drift
+
+The framework uses a **rolling baseline**: every successful comparison updates the baseline to the current run's tensors. This means the reference shifts forward each day. While individual day-to-day diffs stay within the configured threshold, tiny numerical differences can **accumulate over time**, causing the baseline to silently drift away from the original golden values.
+
+**Implications:**
+- The framework detects **regressions** (a sudden, large numerical change between consecutive runs), not **absolute accuracy** relative to a fixed reference.
+- Over weeks or months, the cumulative drift may become significant enough to mask a real regression that happened gradually, or to cause a false-positive failure when the drift eventually crosses the threshold.
+
+**Mitigation strategies (not yet implemented):**
+- Periodically re-establish a fresh anchor baseline from a known-good reference commit.
+- Track the cumulative drift in the manifest metadata and alert when it exceeds a long-term budget.
+- Compare against a fixed "epoch" baseline in addition to the rolling one.
+
+### No local-only mode
+
+The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token.
+
+---
+
+## File Reference
+
+| File | Role |
+|------|------|
+| `test/registered/debug_utils/test_nightly_precision_regression.py` | Main test — server lifecycle, dump, compare, report |
+| `python/sglang/test/precision_baseline_store.py` | HF dataset store — push, fetch, prune baselines |
+| `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine |
+| `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture |
+| `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition |
+| `test/run_suite.py` | Test suite registration (includes `nightly-precision-8-gpu-h200`) |

docs_new/docs/references/overview.mdx

@@ -0,0 +1,13 @@
+---
+title: References
+description: FAQ, environment variables, production metrics, deployment guides, and more.
+---
+
+- [FAQ](./faq)
+- [Environment Variables](./environment_variables)
+- [Production Metrics](./production_metrics)
+- [Production Request Trace](./production_request_trace)
+- [Multi-Node Deployment](./multi_node_deployment/multi_node)
+- [Custom Chat Template](./custom_chat_template)
+- [Frontend Language](./frontend/frontend_tutorial)
+- [Post-Training Integration](./post_training_integration)

docs_new/docs/references/post_training_integration.mdx

@@ -0,0 +1,34 @@
+---
+title: "Post-Training Integration"
+metatags:
+    description: "SGLang post-training: RLHF integration with Miles, slime, AReaL, ROLL, verl, Unsloth, LLaMA Factory."
+---
+SGLang has become the de facto inference backend for modern LLM training frameworks, powering state-of-the-art models across the industry. From GLM-4.6 to Qwen3, leading models leverage SGLang's high-performance inference during reinforcement learning and post-training workflows.
+
+What makes SGLang essential for post-training?
+
+- Open-To-Use Refit Functionality: diverse method for colocate or disaggregate
+- Easy To Postpone Generation: enable partial rollout and dedicated rollout control
+- Fine-Grained Engine Sleep And Wake Up: facilitate maxium-powered rollout and training
+- Training Serving Alignment: ensure the performance consistency in training and serving
+- Load Balancing Router: cache-aware load-balancing for high-throughput rollout
+- Deterministic Inference: ensure zero kl divergence between rollout and training
+
+These capabilities, combined with native integration support across major frameworks, have established SGLang as the infrastructure backbone for modern LLM/VLMs post-training. We also share our latest work in this slide, [Optimizing Large-Scale RL with SGLang](https://gamma.app/docs/Optimizing-RL-with-SGLang-y0kqgj877k34779).
+
+## Adoption
+
+- [**Miles**](https://github.com/radixark/miles): Enterprise-scale RL framework for large MoE models with SGLang-native rollout, speculative training, and production-grade stability
+- [**slime**](https://github.com/THUDM/slime): Post-training framework combining Megatron and SGLang, used to train GLM-4.6
+- [**AReaL**](https://github.com/inclusionAI/AReaL): Fully asynchronous RL system achieving 2.77x speedup with SGLang backend for continuous rollout generation
+- [**ROLL**](https://github.com/alibaba/ROLL): ROLL is an efficient and user-friendly RL library designed for Large Language Models utilizing Large Scale GPU resources
+- [**verl**](https://github.com/volcengine/verl): Full-stack RLHF framework supporting PPO, GRPO, and ReMax with modular SGLang integration
+- [**Unsloth**](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide): 2x faster fine-tuning with optimized kernels, deploys seamlessly with SGLang inference
+- [**LLaMA Factory**](https://github.com/hiyouga/LLaMA-Factory): Unified framework for training 100+ LLMs with LoRA, QLoRA, and full fine-tuning methods
+- [**Tunix**](https://github.com/google/tunix): Google's JAX-native library for LLM post-training with SFT, DPO, PPO, and GRPO support
+- [**RL2**](https://github.com/ChenmienTan/RL2): Ray Less Reinforcement Learning, a concise library of post-training for large language models
+
+
+## Collaboration
+
+Due to the privacy of the design parternes, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **rl_team@lmsys.org** for partnerships, integration guidance, and custom feature development.

docs_new/docs/references/production_metrics.mdx

@@ -0,0 +1,276 @@
+---
+title: "Production Metrics"
+metatags:
+    description: "SGLang Prometheus metrics: TTFT, TPOT, throughput, cache hit rate. Grafana dashboard setup guide."
+---
+SGLang exposes the following metrics via Prometheus. You can enable it by adding `--enable-metrics` when you launch the server.
+
+An example of the monitoring dashboard is available in [examples/monitoring/grafana.json](https://github.com/sgl-project/sglang/blob/main/examples/monitoring/grafana/dashboards/json/sglang-dashboard.json).
+
+Here is an example of the metrics:
+
+```text Output
+$ curl http://localhost:30000/metrics
+# HELP sglang:prompt_tokens_total Number of prefill tokens processed.
+# TYPE sglang:prompt_tokens_total counter
+sglang:prompt_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.128902e+06
+# HELP sglang:generation_tokens_total Number of generation tokens processed.
+# TYPE sglang:generation_tokens_total counter
+sglang:generation_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.557572e+06
+# HELP sglang:token_usage The token usage
+# TYPE sglang:token_usage gauge
+sglang:token_usage{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.28
+# HELP sglang:cache_hit_rate The cache hit rate
+# TYPE sglang:cache_hit_rate gauge
+sglang:cache_hit_rate{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.007507552643049313
+# HELP sglang:time_to_first_token_seconds Histogram of time to first token in seconds.
+# TYPE sglang:time_to_first_token_seconds histogram
+sglang:time_to_first_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2.3518979474117756e+06
+sglang:time_to_first_token_seconds_bucket{le="0.001",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
+sglang:time_to_first_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
+sglang:time_to_first_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
+sglang:time_to_first_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
+sglang:time_to_first_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0
+sglang:time_to_first_token_seconds_bucket{le="0.06",model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0
+sglang:time_to_first_token_seconds_bucket{le="0.08",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:time_to_first_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:time_to_first_token_seconds_bucket{le="0.25",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:time_to_first_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:time_to_first_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:time_to_first_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 27.0
+sglang:time_to_first_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0
+sglang:time_to_first_token_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 314.0
+sglang:time_to_first_token_seconds_bucket{le="7.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 941.0
+sglang:time_to_first_token_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1330.0
+sglang:time_to_first_token_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1970.0
+sglang:time_to_first_token_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2326.0
+sglang:time_to_first_token_seconds_bucket{le="25.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2417.0
+sglang:time_to_first_token_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2513.0
+sglang:time_to_first_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0
+sglang:time_to_first_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0
+# HELP sglang:e2e_request_latency_seconds Histogram of End-to-end request latency in seconds
+# TYPE sglang:e2e_request_latency_seconds histogram
+sglang:e2e_request_latency_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.116093850019932e+06
+sglang:e2e_request_latency_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
+sglang:e2e_request_latency_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="0.8",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="1.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="2.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
+sglang:e2e_request_latency_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.0
+sglang:e2e_request_latency_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 10.0
+sglang:e2e_request_latency_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11.0
+sglang:e2e_request_latency_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 14.0
+sglang:e2e_request_latency_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 247.0
+sglang:e2e_request_latency_seconds_bucket{le="40.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 486.0
+sglang:e2e_request_latency_seconds_bucket{le="50.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 845.0
+sglang:e2e_request_latency_seconds_bucket{le="60.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1513.0
+sglang:e2e_request_latency_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0
+sglang:e2e_request_latency_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0
+# HELP sglang:time_per_output_token_seconds Histogram of time per output token in seconds.
+# TYPE sglang:time_per_output_token_seconds histogram
+sglang:time_per_output_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 866964.5791549598
+sglang:time_per_output_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0
+sglang:time_per_output_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 73.0
+sglang:time_per_output_token_seconds_bucket{le="0.015",model_name="meta-llama/Llama-3.1-8B-Instruct"} 382.0
+sglang:time_per_output_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 593.0
+sglang:time_per_output_token_seconds_bucket{le="0.025",model_name="meta-llama/Llama-3.1-8B-Instruct"} 855.0
+sglang:time_per_output_token_seconds_bucket{le="0.03",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1035.0
+sglang:time_per_output_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1815.0
+sglang:time_per_output_token_seconds_bucket{le="0.05",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11685.0
+sglang:time_per_output_token_seconds_bucket{le="0.075",model_name="meta-llama/Llama-3.1-8B-Instruct"} 433413.0
+sglang:time_per_output_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.950195e+06
+sglang:time_per_output_token_seconds_bucket{le="0.15",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.039435e+06
+sglang:time_per_output_token_seconds_bucket{le="0.2",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.171662e+06
+sglang:time_per_output_token_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.266055e+06
+sglang:time_per_output_token_seconds_bucket{le="0.4",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.296752e+06
+sglang:time_per_output_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.312226e+06
+sglang:time_per_output_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.339675e+06
+sglang:time_per_output_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.357747e+06
+sglang:time_per_output_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.389414e+06
+sglang:time_per_output_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.400757e+06
+sglang:time_per_output_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.400757e+06
+# HELP sglang:func_latency_seconds Function latency in seconds
+# TYPE sglang:func_latency_seconds histogram
+sglang:func_latency_seconds_sum{name="generate_request"} 4.514771912145079
+sglang:func_latency_seconds_bucket{le="0.05",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.07500000000000001",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.1125",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.16875",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.253125",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.3796875",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.56953125",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="0.8542968750000001",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="1.2814453125",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="1.9221679687500002",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="2.8832519531250003",name="generate_request"} 14006.0
+sglang:func_latency_seconds_bucket{le="4.3248779296875",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="6.487316894531251",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="9.730975341796876",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="14.596463012695313",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="21.89469451904297",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="32.84204177856446",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="49.26306266784668",name="generate_request"} 14007.0
+sglang:func_latency_seconds_bucket{le="+Inf",name="generate_request"} 14007.0
+sglang:func_latency_seconds_count{name="generate_request"} 14007.0
+# HELP sglang:num_running_reqs The number of running requests
+# TYPE sglang:num_running_reqs gauge
+sglang:num_running_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 162.0
+# HELP sglang:num_used_tokens The number of used tokens
+# TYPE sglang:num_used_tokens gauge
+sglang:num_used_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 123859.0
+# HELP sglang:gen_throughput The generate throughput (token/s)
+# TYPE sglang:gen_throughput gauge
+sglang:gen_throughput{model_name="meta-llama/Llama-3.1-8B-Instruct"} 86.50814177726902
+# HELP sglang:num_queue_reqs The number of requests in the waiting queue
+# TYPE sglang:num_queue_reqs gauge
+sglang:num_queue_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2826.0
+# HELP sglang:spec_num_steps Currently active speculative_num_steps.
+# TYPE sglang:spec_num_steps gauge
+sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0
+# HELP sglang:spec_num_draft_tokens Currently active speculative_num_draft_tokens (decouples from steps under topk>1).
+# TYPE sglang:spec_num_draft_tokens gauge
+sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0
+```
+
+## Setup Guide
+
+This section describes how to set up the monitoring stack (Prometheus + Grafana) provided in the `examples/monitoring` directory.
+
+### Prerequisites
+
+- Docker and Docker Compose installed
+- SGLang server running with metrics enabled
+
+### Usage
+
+1.  **Start your SGLang server with metrics enabled:**
+
+    ```bash Command
+    python -m sglang.launch_server \
+      --model-path <your_model_path> \
+      --port 30000 \
+      --enable-metrics \
+      --enable-mfu-metrics
+    ```
+    Replace `<your_model_path>` with the actual path to your model (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct`). Ensure the server is accessible from the monitoring stack (you might need `--host 0.0.0.0` if running in Docker). By default, the metrics endpoint will be available at `http://<sglang_server_host>:30000/metrics`.
+
+2.  **Navigate to the monitoring example directory:**
+    ```bash Command
+    cd examples/monitoring
+    ```
+
+3.  **Start the monitoring stack:**
+    ```bash Command
+    docker compose up -d
+    ```
+    This command will start Prometheus and Grafana in the background.
+
+4.  **Access the monitoring interfaces:**
+    *   **Grafana:** Open your web browser and go to [http://localhost:3000](http://localhost:3000).
+    *   **Prometheus:** Open your web browser and go to [http://localhost:9090](http://localhost:9090).
+
+5.  **Log in to Grafana:**
+    *   Default Username: `admin`
+    *   Default Password: `admin`
+    You will be prompted to change the password upon your first login.
+
+6.  **View the Dashboard:**
+    The SGLang dashboard is pre-configured and should be available automatically. Navigate to `Dashboards` -> `Browse` -> `SGLang Monitoring` folder -> `SGLang Dashboard`.
+
+### Troubleshooting
+
+*   **Port Conflicts:** If you encounter errors like "port is already allocated," check if other services (including previous instances of Prometheus/Grafana) are using ports `9090` or `3000`. Use `docker ps` to find running containers and `docker stop <container_id>` to stop them, or use `lsof -i :<port>` to find other processes using the ports. You might need to adjust the ports in the `docker-compose.yaml` file if they permanently conflict with other essential services on your system.
+
+To modify Grafana's port to the other one(like 3090) in your Docker Compose file, you need to explicitly specify the port mapping under the grafana service.
+
+    Option 1: Add GF_SERVER_HTTP_PORT to the environment section:
+    ```
+      environment:
+    - GF_AUTH_ANONYMOUS_ENABLED=true
+    - GF_SERVER_HTTP_PORT=3090  # <-- Add this line
+    ```
+    Option 2: Use port mapping:
+    ```
+    grafana:
+      image: grafana/grafana:latest
+      container_name: grafana
+      ports:
+      - "3090:3000"  # <-- Host:Container port mapping
+    ```
+*   **Connection Issues:**
+    *   Ensure both Prometheus and Grafana containers are running (`docker ps`).
+    *   Verify the Prometheus data source configuration in Grafana (usually auto-configured via `grafana/datasources/datasource.yaml`). Go to `Connections` -> `Data sources` -> `Prometheus`. The URL should point to the Prometheus service (e.g., `http://prometheus:9090`).
+    *   Confirm that your SGLang server is running and the metrics endpoint (`http://<sglang_server_host>:30000/metrics`) is accessible *from the Prometheus container*. If SGLang is running on your host machine and Prometheus is in Docker, use `host.docker.internal` (on Docker Desktop) or your machine's network IP instead of `localhost` in the `prometheus.yaml` scrape configuration.
+*   **No Data on Dashboard:**
+    *   Generate some traffic to your SGLang server to produce metrics. For example, run a benchmark:
+        ```bash Command
+        python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompts 100 --random-input 128 --random-output 128
+        ```
+    *   Check the Prometheus UI (`http://localhost:9090`) under `Status` -> `Targets` to see if the SGLang endpoint is being scraped successfully.
+    *   Verify the `model_name` and `instance` labels in your Prometheus metrics match the variables used in the Grafana dashboard. You might need to adjust the Grafana dashboard variables or the labels in your Prometheus configuration.
+
+### Configuration Files
+
+The monitoring setup is defined by the following files within the `examples/monitoring` directory:
+
+*   `docker-compose.yaml`: Defines the Prometheus and Grafana services.
+*   `prometheus.yaml`: Prometheus configuration, including scrape targets.
+*   `grafana/datasources/datasource.yaml`: Configures the Prometheus data source for Grafana.
+*   `grafana/dashboards/config/dashboard.yaml`: Tells Grafana to load dashboards from the specified path.
+*   `grafana/dashboards/json/sglang-dashboard.json`: The actual Grafana dashboard definition in JSON format.
+
+You can customize the setup by modifying these files. For instance, you might need to update the `static_configs` target in `prometheus.yaml` if your SGLang server runs on a different host or port.
+
+#### Check if the metrics are being collected
+
+Run:
+```text Output
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name random \
+  --num-prompts 3000 \
+  --random-input 1024 \
+  --random-output 1024 \
+  --random-range-ratio 0.5
+```
+
+to generate some requests.
+
+Then you should be able to see the metrics in the Grafana dashboard.
+
+## Estimated Performance Metrics (MFU-related)
+
+SGLang exports the following estimated per-GPU counters that can be used to derive
+Model FLOPs Utilization (MFU)-related signals:
+
+- `sglang:estimated_flops_per_gpu_total`: Estimated floating-point operations.
+- `sglang:estimated_read_bytes_per_gpu_total`: Estimated bytes read from memory.
+- `sglang:estimated_write_bytes_per_gpu_total`: Estimated bytes written to memory.
+
+These metrics are available when both `--enable-metrics` and
+`--enable-mfu-metrics` are enabled.
+
+These are cumulative counters. Use Prometheus `rate(...)` to get per-second values.
+
+### PromQL examples
+
+Average TFLOPS per GPU:
+
+```promql
+rate(sglang:estimated_flops_per_gpu_total[1m]) / 1e12
+```
+
+Average estimated memory bandwidth in GB/s:
+
+```promql
+(rate(sglang:estimated_read_bytes_per_gpu_total[1m]) +
+ rate(sglang:estimated_write_bytes_per_gpu_total[1m])) / 1e9
+```
+
+### Notes
+
+- These metrics are estimates intended for observability and trend analysis.
+- Estimated memory bytes reflect modeled traffic and are not a direct hardware
+  counter from GPU profilers.

docs_new/docs/references/production_request_trace.mdx

@@ -0,0 +1,141 @@
+---
+title: "Production Request Tracing"
+metatags:
+    description: "SGLang OpenTelemetry tracing: Jaeger visualization, trace context propagation, PD disaggregation support."
+---
+SGLang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server.
+
+You can find example screenshots of the visualization in https://github.com/sgl-project/sglang/issues/8965.
+
+## Setup Guide
+This section explains how to configure the request tracing and export the trace data.
+1. Install the required packages and tools
+    * install Docker and Docker Compose
+    * install the dependencies
+    ```bash Command
+    # enter the SGLang root directory
+    pip install -e "python[tracing]"
+
+    # or manually install the dependencies using pip
+    pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-grpc
+    ```
+
+2. Launch OpenTelemetry collector and Jaeger
+    ```bash Command
+    docker compose -f examples/monitoring/tracing_compose.yaml up -d
+    ```
+
+3. Start your SGLang server with tracing enabled
+    ```bash Command
+    # set env variables
+    export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
+    export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
+    # start the prefill and decode server
+    python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
+    # start the model-gate-way
+    python -m sglang_router.launch_router --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
+    ```
+
+    Replace `0.0.0.0:4317` with the actual endpoint of the OpenTelemetry collector. If you launched the openTelemetry collector with tracing_compose.yaml, the default receiving port is 4317.
+
+    To use the HTTP/protobuf span exporter, set the following environment variable and point to an HTTP endpoint, for example, `http://0.0.0.0:4318/v1/traces`.
+    ```bash Command
+    export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
+    ```
+
+
+4. Raise some requests
+5. Observe whether trace data is being exported
+    * Access port 16686 of Jaeger using a web browser to visualize the request traces.
+    * The OpenTelemetry Collector also exports trace data in JSON format to /tmp/otel_trace.json. In a follow-up patch, we will provide a tool to convert this data into a Perfetto-compatible format, enabling visualization of requests in the Perfetto UI.
+
+6. Dynamically adjust trace level
+    The trace level accepts configurable values from `0` to `3`. The meanings of different trace level values are as follows:
+    ```
+    0: disable tracing
+    1: Trace important slices
+    2: Trace all slices except nested ones
+    3: Trace all slices (default)
+    ```
+    **At startup** — set `SGLANG_TRACE_LEVEL` before launching the server:
+    ```bash Command
+    SGLANG_TRACE_LEVEL=2 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other options>
+    ```
+
+    **At runtime** — dynamically adjust via HTTP API without restarting:
+    ```bash Command
+    curl http://0.0.0.0:30000/set_trace_level?level=2
+    ```
+    Replace `0.0.0.0:30000` with your actual server address, and replace `level=2` with the level you want to set.
+
+    **Note**: You must set the parameter `--enable-trace`; otherwise, the trace capability will not be enabled regardless of any dynamic adjustments to the trace level.
+
+## How to add Tracing for slices you're interested in?(API introduction)
+We have already inserted instrumentation points in the tokenizer and scheduler main threads. If you wish to trace additional request execution segments or perform finer-grained tracing, please use the APIs from the tracing package as described below.
+
+**All of the following implementations are done in python/sglang/srt/observability/req_time_stats.py. If you want to add another slice, please do it here.**
+
+1. Initialization
+
+    Every process involved in tracing during the initialization phase should execute:
+    ```python Example
+    process_tracing_init(otlp_traces_endpoint, server_name)
+    ```
+    The otlp_traces_endpoint is obtained from the arguments, and you can set server_name freely, but it should remain consistent across all processes.
+
+    Every thread involved in tracing during the initialization phase should execute:
+    ```python Example
+    trace_set_thread_info("thread label", tp_rank, dp_rank)
+    ```
+    The "thread label" can be regarded as the name of the thread, used to distinguish different threads in the visualization view.
+
+2. Create a trace context for a request
+    Each request needs to call `TraceReqContext()` to initialize a request context, which is used to generate slice spans and record request stage info. You can either store it within the request object or maintain it as a global variable.
+
+3. Mark the beginning and end of a request
+    ```
+    trace_ctx.trace_req_start().
+    trace_ctx.trace_req_finish()
+    ```
+    trace_req_start() and trace_req_finish() must be called within the same process, for example, in the tokenizer.
+
+4. Add tracing for a slice
+
+    * Add slice tracing normally:
+        ```python Example
+        trace_ctx.trace_slice_start(RequestStage.TOKENIZER.stage_name)
+        trace_ctx.trace_slice_end(RequestStage.TOKENIZER.stage_name)
+
+        or
+        trace_ctx.trace_slice(slice: TraceSliceContext)
+        ```
+
+    - The end of the last slice in a thread must be marked with thread_finish_flag=True, or explicitly call trace_ctx.abort(); otherwise, the thread's span will not be properly generated.
+        ```python Example
+        trace_ctx.slice_end(RequestStage.D.stage_name, thread_finish_flag = True)
+        trace_ctx.abort()
+        ```
+
+5. When the request execution flow transfers to another thread, the thread context needs to be explicitly rebuilt.
+    - receiver: Execute the following code after receiving the request via ZMQ
+        ```python Example
+        trace_ctx.rebuild_thread_context()
+        ```
+
+## How to Extend the Tracing Framework to Support Complex Tracing Scenarios
+
+The currently provided tracing package still has potential for further development. If you wish to build more advanced features upon it, you must first understand its existing design principles.
+
+The core of the tracing framework's implementation lies in the design of the span structure and the trace context. To aggregate scattered slices and enable concurrent tracking of multiple requests, we have designed a three-level trace context structure or span structure: `TraceReqContext`, `TraceThreadContext` and `TraceSliceContext`. Their relationship is as follows:
+```
+TraceReqContext (req_id="req-123")
+├── TraceThreadContext(thread_label="scheduler", tp_rank=0)
+|     └── TraceSliceContext(slice_name="prefill")
+|
+└── TraceThreadContext(thread_label="scheduler", tp_rank=1)
+      └── TraceSliceContext(slice_name="prefill")
+```
+
+Each traced request maintains a global `TraceReqContext` and creates a corresponding request span. For every thread that processes the request, a `TraceThreadContext` is recorded and a thread span is created. The `TraceThreadContext` is nested within the `TraceReqContext`, and each currently traced code slice—potentially nested—is stored in its associated `TraceThreadContext`.
+
+In addition to the above hierarchy, each slice also records its previous slice via Span.add_link(), which can be used to trace the execution flow.

docs_new/docs/references/torch_compile_cache.mdx

@@ -0,0 +1,16 @@
+---
+title: "Enabling cache for torch.compile"
+metatags:
+    description: "SGLang torch.compile cache: TORCHINDUCTOR_CACHE_DIR for faster deployment across multiple machines."
+---
+SGLang uses `max-autotune-no-cudagraphs` mode of torch.compile. The auto-tuning can be slow.
+If you want to deploy a model on many different machines, you can ship the torch.compile cache to these machines and skip the compilation steps.
+
+This is based on https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html
+
+
+1. Generate the cache by setting TORCHINDUCTOR_CACHE_DIR and running the model once.
+```text Output
+TORCHINDUCTOR_CACHE_DIR=/root/inductor_root_cache python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --enable-torch-compile
+```
+2. Copy the cache folder to other machines and launch the server with `TORCHINDUCTOR_CACHE_DIR`.