wehub-resource-sync
296f9921e4
Lint / lint (push) Has been cancelled
PR Test (Arm64) / check-changes (push) Has been cancelled
PR Test (NPU) / check-changes (push) Has been cancelled
PR Test (NPU) / set-image-config (push) Has been cancelled
PR Test (sgl-router) / gate (push) Has been cancelled
PR Test (Xeon) / check-changes (push) Has been cancelled
PR Test (XPU) / check-changes (push) Has been cancelled
PR Test (Arm64) / pr-gate (push) Has been cancelled
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Has been cancelled
PR Test (XPU) / pr-gate (push) Has been cancelled
PR Test (XPU) / wait-for-stage-a (push) Has been cancelled
PR Test (XPU) / stage-b-test-1-gpu-xpu (push) Has been cancelled
PR Test (XPU) / finish (push) Has been cancelled
PR Test (Arm64) / build-test (push) Has been cancelled
PR Test (NPU) / pr-gate (push) Has been cancelled
PR Test (NPU) / stage-b-test-1-npu-a2 (0) (push) Has been cancelled
PR Test (NPU) / stage-b-test-1-npu-a2 (1) (push) Has been cancelled
PR Test (NPU) / stage-b-test-2-npu-a2 (0) (push) Has been cancelled
PR Test (NPU) / stage-b-test-2-npu-a2 (1) (push) Has been cancelled
PR Test (NPU) / stage-b-test-4-npu-a3 (push) Has been cancelled
PR Test (NPU) / stage-b-test-16-npu-a3 (push) Has been cancelled
PR Test (NPU) / multimodal-gen-test-1-npu-a3 (push) Has been cancelled
PR Test (NPU) / multimodal-gen-test-2-npu-a3 (push) Has been cancelled
PR Test (NPU) / single-node-poc (map[name:qwen3_6_27b_w8a8_1p_in64k_out1k_50ms runner:linux-aarch64-a3-2 test_case:test/registered/ascend/performance/qwen3_6_27b/test_npu_qwen3_6_27b_w8a8_1p_in64k_out1k_50ms.py test_type:perf]) (push) Has been cancelled
PR Test (NPU) / pr-test-npu-finish (push) Has been cancelled
PR Test (sgl-router) / tier-1 — lint (push) Has been cancelled
PR Test (sgl-router) / tier-2 — build + test (push) Has been cancelled
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Has been cancelled
PR Test (sgl-router) / tier-3 — k8s integration (push) Has been cancelled
PR Test (sgl-router) / tier-3 — e2e (push) Has been cancelled
PR Test (sgl-router) / finish (push) Has been cancelled
PR Test (Xeon) / pr-gate (push) Has been cancelled
PR Test (Xeon) / build-test (, xeon-gnr, base-b-test-cpu) (push) Has been cancelled
102 lines
12 KiB
Markdown
102 lines
12 KiB
Markdown
<!-- WEHUB_ZH_README -->
|
||
> [!NOTE]
|
||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||
> [English](./README.en.md) · [原始项目](https://github.com/sgl-project/sglang) · [上游 README](https://github.com/sgl-project/sglang/blob/HEAD/README.md)
|
||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||
|
||
<div align="center" id="sglangtop">
|
||
<img src="https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png" alt="logo" width="400" margin="10px"></img>
|
||
|
||
[](https://pypi.org/project/sglang)
|
||
|
||
[](https://github.com/sgl-project/sglang/tree/main/LICENSE)
|
||
[](https://github.com/sgl-project/sglang/issues)
|
||
[](https://github.com/sgl-project/sglang/issues)
|
||
[](https://deepwiki.com/sgl-project/sglang)
|
||
|
||
</div>
|
||
|
||
--------------------------------------------------------------------------------
|
||
|
||
<p align="center">
|
||
<a href="https://lmsys.org/blog/"><b>博客</b></a> |
|
||
<a href="https://docs.sglang.io/"><b>文档</b></a> |
|
||
<a href="https://roadmap.sglang.io/"><b>路线图</b></a> |
|
||
<a href="https://slack.sglang.io/"><b>加入 Slack</b></a> |
|
||
<a href="https://meet.sglang.io/"><b>每周开发会议</b></a> |
|
||
<a href="https://github.com/sgl-project/sgl-learning-materials?tab=readme-ov-file#slides"><b>幻灯片</b></a>
|
||
</p>
|
||
|
||
## 新闻
|
||
- [2026/06] 🔥 下一代推测解码（Speculative Decoding）：DFlash 与 Spec V2（[博客](https://lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/)).
|
||
- [2026/04] 🔥 DeepSeek-V4 首日上线：从 SGLang 与 Miles 实现快速推理到可验证强化学习（[博客](https://lmsys.org/blog/2026-04-25-deepseek-v4/)).
|
||
- [2026/06] SGLang 为最新开源模型提供首日（day-0）支持（[Nemotron 3 Ultra](https://lmsys.org/blog/2026-06-04-nvidia-run-nemotron-3-ultra/), [Nemotron 3 Super](https://lmsys.org/blog/2026-03-11-run-nvidia-nemotron-3-super/), [Higgs Audio v3 TTS](https://lmsys.org/blog/2026-06-04-higgs-audio-v3-tts/)).
|
||
- [2026/02] 🔥 在 NVIDIA GB300 NVL72 上借助 SGLang 解锁 25 倍推理性能（[博客](https://lmsys.org/blog/2026-02-20-gb300-inferencex/)).
|
||
- [2026/01] SGLang Diffusion 加速视频与图像生成（[博客](https://lmsys.org/blog/2026-01-16-sglang-diffusion/)).
|
||
- [2025/12] SGLang 为最新开源模型提供首日（day-0）支持（[MiMo-V2-Flash](https://lmsys.org/blog/2025-12-16-mimo-v2-flash/), [Nemotron 3 Nano](https://lmsys.org/blog/2025-12-15-run-nvidia-nemotron-3-nano/), [Mistral Large 3](https://github.com/sgl-project/sglang/pull/14213), [LLaDA 2.0 Diffusion LLM](https://lmsys.org/blog/2025-12-19-diffusion-llm/), [MiniMax M2](https://lmsys.org/blog/2025-11-04-miminmax-m2/)).
|
||
- [2025/10] SGLang 现可通过 SGLang-Jax 后端原生运行于 TPU（[博客](https://lmsys.org/blog/2025-10-29-sglang-jax/)).
|
||
|
||
<details>
|
||
<summary>更多</summary>
|
||
|
||
- [2025/09] 在 GB200 NVL72 上部署 DeepSeek，采用 PD 与大规模 EP（第二部分）：Prefill 提升 3.8 倍，Decode 吞吐量提升 4.8 倍（[博客](https://lmsys.org/blog/2025-09-25-gb200-part-2/)).
|
||
- [2025/09] SGLang 首日支持带稀疏注意力（Sparse Attention）的 DeepSeek-V3.2（[博客](https://lmsys.org/blog/2025-09-29-deepseek-V32/)).
|
||
- [2025/08] SGLang x AMD SF 线下聚会（8/22）：GPU 动手工作坊、AMD/xAI/SGLang 技术分享与社交交流（[路线图](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_sglang_roadmap.pdf), [大规模 EP](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_sglang_ep.pdf), [亮点](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_highlights.pdf), [AITER/MoRI](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_aiter_mori.pdf), [Wave](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_wave.pdf)).
|
||
|
||
- [2025/11] SGLang Diffusion 加速视频与图像生成（[博客](https://lmsys.org/blog/2025-11-07-sglang-diffusion/)).
|
||
- [2025/10] PyTorch Conference 2025 SGLang 演讲（[幻灯片](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/sglang_pytorch_2025.pdf)).
|
||
- [2025/10] SGLang x Nvidia SF 线下聚会（10/2）（[回顾](https://x.com/lmsysorg/status/1975339501934510231)).
|
||
- [2025/08] SGLang 为 OpenAI gpt-oss 模型提供首日（day-0）支持（[说明](https://github.com/sgl-project/sglang/issues/8833))
|
||
- [2025/06] SGLang 作为每日驱动数万亿 token 的高性能服务基础设施，荣获 a16z 第三批开源 AI 资助（[a16z 博客](https://a16z.com/advancing-open-source-ai-through-benchmarks-and-bold-experimentation/)).
|
||
- [2025/05] 在 96 张 H100 GPU 上通过 PD 分离与大规模专家并行（Expert Parallelism）部署 DeepSeek（[博客](https://lmsys.org/blog/2025-05-05-large-scale-ep/)).
|
||
- [2025/06] 在 GB200 NVL72 上部署 DeepSeek，采用 PD 与大规模 EP（第一部分）：Decode 吞吐量提升 2.7 倍（[博客](https://lmsys.org/blog/2025-06-16-gb200-part-1/)).
|
||
- [2025/03] 在 AMD Instinct MI300X 上为 DeepSeek-R1 推理加速（[AMD 博客](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html))
|
||
- [2025/03] SGLang 加入 PyTorch 生态：高效 LLM 服务引擎（[PyTorch 博客](https://pytorch.org/blog/sglang-joins-pytorch/))
|
||
- [2025/02] 在 AMD Instinct™ MI300X GPU 上解锁 DeepSeek-R1 推理性能（[AMD 博客](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1_Perf/README.html))
|
||
- [2025/01] SGLang 在 NVIDIA 与 AMD GPU 上为 DeepSeek V3/R1 模型提供首日支持，并包含 DeepSeek 专属优化。（[说明](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3), [AMD 博客](https://www.amd.com/en/developer/resources/technical-articles/amd-instinct-gpus-power-deepseek-v3-revolutionizing-ai-development-with-sglang.html), [10+ 其他公司](https://x.com/lmsysorg/status/1887262321636221412))
|
||
- [2024/12] v0.4 发布：零开销批处理调度器、缓存感知负载均衡器、更快的结构化输出（[博客](https://lmsys.org/blog/2024-12-04-sglang-v0-4/)).
|
||
- [2024/10] 首届 SGLang 线上聚会（[幻灯片](https://github.com/sgl-project/sgl-learning-materials?tab=readme-ov-file#the-first-sglang-online-meetup)).
|
||
- [2024/09] v0.3 发布：DeepSeek MLA 快 7 倍、torch.compile 快 1.5 倍、多图/视频 LLaVA-OneVision（[博客](https://lmsys.org/blog/2024-09-04-sglang-v0-3/)).
|
||
- [2024/07] v0.2 发布：借助 SGLang Runtime 更快服务 Llama3（对比 TensorRT-LLM、vLLM）（[博客](https://lmsys.org/blog/2024-07-25-sglang-llama3/)).
|
||
- [2024/02] SGLang 借助压缩有限状态机实现 **JSON 解码快 3 倍**（[博客](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
|
||
- [2024/01] SGLang 借助 RadixAttention 实现最高 **5 倍更快推理**（[博客](https://lmsys.org/blog/2024-01-17-sglang/)).
|
||
- [2024/01] SGLang 为官方 **LLaVA v1.6** 发布演示提供推理服务（[用法](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).
|
||
|
||
</details>
|
||
|
||
## 简介
|
||
SGLang 是一个面向大语言模型（LLM）与多模态模型的高性能服务框架。
|
||
它旨在在从单 GPU 到大型分布式集群的广泛部署场景中，提供低延迟、高吞吐量的推理能力。
|
||
其核心特性包括：
|
||
|
||
- **快速运行时（Fast Runtime）**：通过 RadixAttention 实现前缀缓存（prefix caching）、零开销 CPU 调度器、prefill-decode 分离（prefill-decode disaggregation）、推测解码（speculative decoding）、连续批处理（continuous batching）、分页注意力（paged attention）、张量/流水线/专家/数据并行（tensor/pipeline/expert/data parallelism）、结构化输出（structured outputs）、分块 prefill（chunked prefill）、量化（FP4/FP8/INT4/AWQ/GPTQ）以及多 LoRA 批处理，提供高效服务。
|
||
- **广泛的模型支持（Broad Model Support）**：支持多种语言模型（Llama、Qwen、DeepSeek、Kimi、GLM、GPT、Gemma、Mistral 等）、嵌入模型（e5-mistral、gte、mcdse）、奖励模型（Skywork）以及扩散模型（WAN、Qwen-Image），并易于扩展以接入新模型。兼容大多数 Hugging Face 模型与 OpenAI API。
|
||
- **广泛的硬件支持（Extensive Hardware Support）**：可运行于 NVIDIA GPU（GB200/B300/H100/A100/Spark/5090）、AMD GPU（MI355/MI300）、Intel Xeon CPU、Google TPU、Ascend NPU 等。
|
||
- **活跃的社区（Active Community）**：SGLang 是开源项目，拥有活跃社区与广泛的行业采用，全球范围内驱动超过 400,000 张 GPU。
|
||
- **强化学习与后训练骨干（RL & Post-Training Backbone）**：SGLang 是经实战验证的 rollout 后端，已用于训练众多前沿模型，具备原生 RL 集成，并被知名后训练框架广泛采用，例如 [**AReaL**](https://github.com/inclusionAI/AReaL), [**Miles**](https://github.com/radixark/miles), [**slime**](https://github.com/THUDM/slime), [**Tunix**](https://github.com/google/tunix), [**verl**](https://github.com/volcengine/verl) 等。
|
||
|
||
## 入门指南
|
||
- [安装 SGLang](https://docs.sglang.io/get_started/install.html)
|
||
- [快速开始](https://docs.sglang.io/basic_usage/send_request.html)
|
||
- [后端教程](https://docs.sglang.io/basic_usage/openai_api_completions.html)
|
||
- [前端教程](https://docs.sglang.io/references/frontend/frontend_tutorial.html)
|
||
- [贡献指南](https://docs.sglang.io/developer_guide/contribution_guide.html)
|
||
|
||
## 基准测试与性能
|
||
详见发布博客：[v0.2 博客](https://lmsys.org/blog/2024-07-25-sglang-llama3/), [v0.3 博客](https://lmsys.org/blog/2024-09-04-sglang-v0-3/), [v0.4 博客](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), [大规模专家并行（Large-scale expert parallelism）](https://lmsys.org/blog/2025-05-05-large-scale-ep/), [GB200 机架级并行（GB200 rack-scale parallelism）](https://lmsys.org/blog/2025-09-25-gb200-part-2/), [GB300 长上下文（GB300 long context）](https://lmsys.org/blog/2026-02-19-gb300-longctx/).
|
||
|
||
## 采用与赞助
|
||
SGLang 已大规模部署，每天在生产环境中生成数万亿 token。它受到众多领先企业与机构的信赖与采用，包括 xAI、AMD、NVIDIA、Intel、LinkedIn、Cursor、Oracle Cloud、Google Cloud、Microsoft Azure、AWS、Atlas Cloud、Voltage Park、Nebius、DataCrunch、Novita、InnoMatrix、Modal、MIT、UCLA、华盛顿大学（University of Washington）、Stanford、UC Berkeley、清华大学（Tsinghua University）、Jam & Tea Studios、Baseten 及其他主要科技组织。
|
||
作为一款开源 LLM 推理引擎，SGLang 已成为事实上的行业标准，全球部署规模超过 400,000 块 GPU。
|
||
SGLang 目前由非营利开源组织 [LMSYS](https://lmsys.org/about/). 托管。
|
||
|
||
<img src="https://raw.githubusercontent.com/sgl-project/sgl-learning-materials/refs/heads/main/slides/adoption.png" alt="logo" width="800" margin="10px"></img>
|
||
|
||
## 联系我们
|
||
如有意大规模采用或部署 SGLang，包括技术咨询、赞助机会或合作洽谈，请通过 [sglang@lmsys.org](mailto:sglang@lmsys.org) 与我们联系。
|
||
|
||
长期活跃的 SGLang 贡献者可能有资格获得编程智能体（coding agent）赞助，例如 Cursor、Claude Code 或 OpenAI Codex。请将您最重要的 commits 或 pull requests 发送至 [sglang@lmsys.org](mailto:sglang@lmsys.org)。
|
||
|
||
## 致谢
|
||
我们从以下项目中学习设计思路并复用代码：[Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), 以及 [LMQL](https://github.com/eth-sri/lmql).
|