docs: make Chinese README the default
Book-CI / test (windows-latest) (push) Has been cancelled
Book-CI / test (macos-latest) (push) Has been cancelled
Book-CI / test (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (macos-latest) (push) Has been cancelled
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (windows-latest) (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 10:41:40 +00:00
parent 49dec5faea
commit 2cc4af1a05
+85 -91
View File
@@ -1,3 +1,9 @@
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/kvcache-ai/ktransformers) · [上游 README](https://github.com/kvcache-ai/ktransformers/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
<div align="center">
<p align="center">
@@ -7,110 +13,99 @@
</picture>
</p>
<h3>A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations</h3>
<strong><a href="#-overview">🎯 Overview</a> | <a href="#-inference---high-performance-kt-kernel-serving">🚀 Inference</a> | <a href="#-sft---fine-tuning-with-llama-factory">🎓 SFT</a> | <a href="#-citation">🔥 Citation</a> | <a href="https://github.com/kvcache-ai/ktransformers/issues/1921">🚀 Roadmap(2026Q2)</a> </strong>
<h3>一个用于体验尖端 LLM 推理/微调优化的灵活框架</h3>
<strong><a href="#-概览">🎯 概览</a> | <a href="#-推理---kt-kernel-高性能推理">🚀 推理</a> | <a href="#-sft---llama-factory-微调">🎓 SFT</a> | <a href="#-引用">🔥 引用</a> </strong>
</div>
## 🎯 Overview
## 🎯 概览
KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: [Inference](./kt-kernel/README.md) and [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md).
KTransformers 是一个专注于通过 CPU-GPU 异构计算实现大语言模型高效推理和微调的研究项目。目前两个面向用户的能力都来自 kt-kernel 源码目录:[推理](./kt-kernel/README.md) [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)
## 🔥 Updates
* **June 21, 2026**: MiniMax-M3 Day0 Support! ([Tutorial](./doc/en/kt-kernel/MiniMax-M3-Tutorial.md))
* **June 17, 2026**: GLM-5.2 Day0 Support! ([Tutorial](./doc/en/kt-kernel/GLM-5.2-Tutorial.md))
* **May 6, 2026**: KTransformers at [GOSIM Paris 2026](https://paris2026.gosim.org/zh/schedule/) — "Agentic AI on Edge" track. We'll present KT's inference performance on consumer hardware.
* **May 02, 2026**: DeepSeek-V4-Flash Support! ([Tutorial](./doc/en/DeepSeek-V4-Flash.md))
* **Apr 30, 2026**: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate [Inference](./kt-kernel/README.md) and [SFT Quick Start](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) entry points.
* **Mar 26, 2026**: Support AVX2-only CPU backend for KT-Kernel inference. ([Tutorial](./doc/en/kt-kernel/AVX2-Tutorial.md))
* **Feb 13, 2026**: MiniMax-M2.5 Day0 Support! ([Tutorial](./doc/en/MiniMax-M2.5.md))
* **Feb 12, 2026**: GLM-5 Day0 Support! ([Tutorial](./doc/en/kt-kernel/GLM-5-Tutorial.md))
* **Jan 27, 2026**: Kimi-K2.5 Day0 Support! ([Tutorial](./doc/en/Kimi-K2.5.md)) ([SFT Tutorial](./doc/en/SFT_Installation_Guide_KimiK2.5.md))
* **Jan 22, 2026**: Support [CPU-GPU Expert Scheduling](./doc/en/kt-kernel/experts-sched-Tutorial.md), [Native BF16 and FP8 per channel Precision](./doc/en/kt-kernel/Native-Precision-Tutorial.md) and [AutoDL unified fine-tuning and inference](./doc/zh/【云端低价训推】%20KTransformers%2BAutoDL%2BLlamaFactory:随用随租的低成本超大模型「微调%2B推理」一体化流程.pdf)
* **Dec 24, 2025**: Support Native MiniMax-M2.1 inference. ([Tutorial](./doc/en/kt-kernel/MiniMax-M2.1-Tutorial.md))
* **Dec 22, 2025**: Support RL-DPO fine-tuning with LLaMA-Factory. ([Tutorial](./doc/en/SFT/DPO_tutorial.md))
* **Dec 5, 2025**: Support Native Kimi-K2-Thinking inference ([Tutorial](./doc/en/kt-kernel/Kimi-K2-Thinking-Native.md))
* **Nov 6, 2025**: Support Kimi-K2-Thinking inference ([Tutorial](./doc/en/Kimi-K2-Thinking.md)) and fine-tune ([Tutorial](./doc/en/SFT_Installation_Guide_KimiK2.md))
* **Nov 4, 2025**: KTransformers Fine-Tuning × LLaMA-Factory Integration. ([Tutorial](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md))
* **Oct 27, 2025**: Support Ascend NPU. ([Tutorial](./doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md))
* **Oct 10, 2025**: Integrating into SGLang. ([Roadmap](https://github.com/sgl-project/sglang/issues/11425), [Blog](https://lmsys.org/blog/2025-10-22-KTransformers/))
* **Sept 11, 2025**: Support Qwen3-Next. ([Tutorial](./doc/en/Qwen3-Next.md))
* **Sept 05, 2025**: Support Kimi-K2-0905. ([Tutorial](./doc/en/Kimi-K2.md))
* **July 26, 2025**: Support SmallThinker and GLM4-MoE. ([Tutorial](./doc/en/SmallThinker_and_Glm4moe.md))
* **July 11, 2025**: Support Kimi-K2. ([Tutorial](./doc/en/Kimi-K2.md))
* **June 30, 2025**: Support 3-layer (GPU-CPU-Disk) [prefix cache](./doc/en/prefix_cache.md) reuse.
* **May 14, 2025**: Support Intel Arc GPU ([Tutorial](./doc/en/xpu.md)).
* **Apr 29, 2025**: Support AMX-Int8、 AMX-BF16 and Qwen3MoE ([Tutorial](./doc/en/AMX.md))
* **Apr 9, 2025**: Experimental support for LLaMA 4 models ([Tutorial](./doc/en/llama4.md)).
* **Apr 2, 2025**: Support Multi-concurrency. ([Tutorial](./doc/en/balance-serve.md)).
* **Mar 15, 2025**: Support ROCm on AMD GPU ([Tutorial](./doc/en/ROCm.md)).
* **Mar 5, 2025**: Support unsloth 1.58/2.51 bits weights and [IQ1_S/FP8 hybrid](./doc/en/fp8_kernel.md) weights. Support 139K [Longer Context](./doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel) for DeepSeek-V3 and R1 in 24GB VRAM.
* **Feb 25, 2025**: Support [FP8 GPU kernel](./doc/en/fp8_kernel.md) for DeepSeek-V3 and R1; [Longer Context](./doc/en/DeepseekR1_V3_tutorial.md#v022-longer-context).
* **Feb 15, 2025**: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed +15%, up to 16 Tokens/s), update [docs](./doc/en/DeepseekR1_V3_tutorial.md) and [online books](https://kvcache-ai.github.io/ktransformers/).
* **Feb 10, 2025**: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see [here](./doc/en/DeepseekR1_V3_tutorial.md).
* **Aug 28, 2024**: Decrease DeepseekV2's required VRAM from 21G to 11G.
* **Aug 15, 2024**: Update detailed [tutorial](doc/en/injection_tutorial.md) for injection and multi-GPU.
* **Aug 14, 2024**: Support llamfile as linear backend.
* **Aug 12, 2024**: Support multiple GPU; Support new model: mixtral 8\*7B and 8\*22B; Support q2k, q3k, q5k dequant on gpu.
* **Aug 9, 2024**: Support windows native.
## 🔥 更新
* **2026 年 4 月 30 日**KTransformers v0.6.1 更新 kt-kernel 推理和 SFT 文档,提供独立的[推理](./kt-kernel/README.md)和 [SFT Quick Start](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)入口。
* **2025 年 12 月 5 日**:支持原生 Kimi-K2-Thinking 推理([教程](./doc/en/kt-kernel/Kimi-K2-Thinking-Native.md)
* **2025 年 11 月 6 日**:支持 Kimi-K2-Thinking 推理([教程](./doc/en/Kimi-K2-Thinking.md))和微调([教程](./doc/en/SFT_Installation_Guide_KimiK2.md)
* **2025 年 11 月 4 日**KTransformers 微调 × LLaMA-Factory 集成([教程](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md)
* **2025 年 10 月 27 日**:支持昇腾 NPU[教程](./doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md)
* **2025 年 10 月 10 日**:集成到 SGLang[路线图](https://github.com/sgl-project/sglang/issues/11425)[博客](https://lmsys.org/blog/2025-10-22-KTransformers/)
* **2025 年 9 月 11 日**:支持 Qwen3-Next[教程](./doc/en/Qwen3-Next.md)
* **2025 年 9 月 5 日**:支持 Kimi-K2-0905[教程](./doc/en/Kimi-K2.md)
* **2025 年 7 月 26 日**:支持 SmallThinker 和 GLM4-MoE[教程](./doc/en/SmallThinker_and_Glm4moe.md)
* **2025 年 7 月 11 日**:支持 Kimi-K2[教程](./doc/en/Kimi-K2.md)
* **2025 年 6 月 30 日**:支持 3 层(GPU-CPU-磁盘)[前缀缓存](./doc/en/prefix_cache.md)复用
* **2025 年 5 月 14 日**:支持 Intel Arc GPU[教程](./doc/en/xpu.md)
* **2025 年 4 月 29 日**:支持 AMX-Int8、AMX-BF16 和 Qwen3MoE[教程](./doc/en/AMX.md)
* **2025 年 4 月 9 日**:实验性支持 LLaMA 4 模型([教程](./doc/en/llama4.md)
* **2025 年 4 月 2 日**:支持多并发([教程](./doc/en/balance-serve.md)
* **2025 年 3 月 15 日**:支持 AMD GPU 上的 ROCm[教程](./doc/en/ROCm.md)
* **2025 年 3 月 5 日**:支持 unsloth 1.58/2.51 位权重和 [IQ1_S/FP8 混合](./doc/en/fp8_kernel.md)权重。在 24GB VRAM 中支持 DeepSeek-V3 和 R1 的 139K [更长上下文](./doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel)
* **2025 年 2 月 25 日**:为 DeepSeek-V3 和 R1 支持 [FP8 GPU 内核](./doc/en/fp8_kernel.md)[更长上下文](./doc/en/DeepseekR1_V3_tutorial.md#v022-longer-context)
* **2025 年 2 月 15 日**:更长上下文(24GB VRAM 从 4K 到 8K& 速度稍快(+15%,最高 16 Tokens/s),更新[文档](./doc/en/DeepseekR1_V3_tutorial.md)和[在线手册](https://kvcache-ai.github.io/ktransformers/)
* **2025 年 2 月 10 日**:支持 Deepseek-R1 和 V3 在单 GPU24GB VRAM/多 GPU 和 382GB DRAM 上运行,速度提升高达 3~28 倍。详细案例展示和复现教程请参见[这里](./doc/en/DeepseekR1_V3_tutorial.md)
* **2024 年 8 月 28 日**:将 DeepseekV2 所需的 VRAM 从 21GB 降低到 11GB
* **2024 年 8 月 15 日**:更新了关于注入和多 GPU 的详细[教程](doc/en/injection_tutorial.md)
* **2024 年 8 月 14 日**:支持 llamfile 作为线性后端
* **2024 年 8 月 12 日**:支持多 GPU;支持新模型:mixtral 8\*7B 和 8\*22B;支持 GPU 上的 q2k、q3k、q5k 去量化
* **2024 年 8 月 9 日**:支持 Windows 原生环境
---
## 📦 Capabilities
## 📦 功能入口
### 🚀 [Inference](./kt-kernel/README.md) - High-Performance kt-kernel Serving
### 🚀 [推理](./kt-kernel/README.md) - kt-kernel 高性能推理
CPU-optimized kernel operations for heterogeneous LLM inference.
用于异构 LLM 推理的 CPU 优化内核操作。
<img width="1049" height="593" alt="image" src="https://github.com/user-attachments/assets/68f423da-3f55-4025-bdc9-9ceaa554f00b" />
![image-20251011010558909](./doc/assets/heterogeneous_computing.png)
**主要特性:**
- **AMX/AVX 加速**Intel AMX 和 AVX512/AVX2 优化的内核,用于 INT4/INT8 量化推理
- **MoE 优化**:高效的专家混合推理,具有 NUMA 感知内存管理
- **量化支持**CPU 端 INT4/INT8 量化权重,GPU 端 GPTQ 支持
- **易于集成**:为 SGLang 和其他框架提供简洁的 Python API
**Key Features:**
- **AMX/AVX Acceleration**: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
- **MoE Optimization**: Efficient Mixture-of-Experts inference with NUMA-aware memory management
- **Quantization Support**: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
- **Easy Integration**: Clean Python API for SGLang and other frameworks
**Quick Start:**
**快速开始:**
```bash
cd kt-kernel
pip install .
```
**Use Cases:**
**使用场景:**
- CPU-GPU hybrid inference for large MoE models
- Integration with SGLang for production serving
- Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)
- 大型 MoE 模型的 CPU-GPU 混合推理
- 与 SGLang 集成用于生产服务
- 异构专家放置(热专家在 GPU 上,冷专家在 CPU 上)
**Performance Examples:**
| Model | Hardware Configuration | Total Throughput | Output Throughput |
**性能示例:**
| 模型 | 硬件配置 | 总吞吐量 | 输出吞吐量 |
|-------|------------------------|------------------|-------------------|
| DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens/s | 87.58 tokens/s (8-way concurrency) |
| DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens/s | 87.58 tokens/s8 路并发)|
👉 **[Full Documentation](./kt-kernel/README.md)**
👉 **[完整文档](./kt-kernel/README.md)**
---
### 🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory
### 🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - LLaMA-Factory 微调
KTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.
KTransformers × LLaMA-Factory 集成,面向超大 MoE 模型微调。
![KTransformers SFT](https://raw.githubusercontent.com/kvcache-ai/ktransformers/main/doc/assets/image-20251011010558909.png)
![KTransformers SFT](./doc/assets/image-20251011010558909.png)
**Key Features:**
- **Multi-Backend Support**: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
- **Ultra-Large MoE Support**: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
- **Faster than ZeRO-Offload**: 6-12x training speedup in benchmarked MoE SFT workloads
- **Lower CPU Memory**: About half the CPU memory of the previous KT SFT path in the benchmarked setup
- **LLaMA-Factory Integration**: Seamless integration with popular fine-tuning framework
**主要特性:**
- **多后端支持**: CPU/GPU 混合微调,支持 INT8/INT4 量化
- **超大 MoE 支持**: 在有限 GPU 内存下微调 DeepSeek-V3/R1 等模型
- **相对 ZeRO-Offload 加速**: 在基准 MoE SFT 任务中训练速度提升 6-12 倍
- **降低 CPU 内存**: 相比上一版 KT SFT 路径,基准配置下 CPU 内存约降至 1/2
- **LLaMA-Factory 集成**: 与流行微调框架无缝集成
| Model | GPU Memory | Training Speed | Hardware |
| 模型 | GPU 内存 | 训练速度 | 硬件 |
|-------|------------|----------------|----------|
| DeepSeek-V3 | ~80GB total | 3.7 it/s | 4x RTX 4090 |
| DeepSeek-R1 | ~80GB total | 3.7 it/s | 4x RTX 4090 |
| Qwen3-30B-A3B | ~24GB total | 8+ it/s | 1x RTX 4090 |
| DeepSeek-V3 | ~80GB 总计 | 3.7 it/s | 4x RTX 4090 |
| DeepSeek-R1 | ~80GB 总计 | 3.7 it/s | 4x RTX 4090 |
| Qwen3-30B-A3B | ~24GB 总计 | 8+ it/s | 1x RTX 4090 |
**Quick Start:**
**快速开始:**
```bash
cd /path/to/LLaMA-Factory
pip install -e .
@@ -122,13 +117,13 @@ CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
```
👉 **[Quick Start →](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)**
👉 **[Full Documentation](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md)**
👉 **[完整文档](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md)**
---
## 🔥 Citation
## 🔥 引用
If you use KTransformers in your research, please cite our paper:
如果您在研究中使用了 KTransformers,请引用我们的论文:
```bibtex
@inproceedings{10.1145/3731569.3764843,
@@ -139,25 +134,24 @@ If you use KTransformers in your research, please cite our paper:
}
```
## 👥 Contributors & Team
## 👥 贡献者与团队
Developed and maintained by:
- [MADSys Lab](https://madsys.cs.tsinghua.edu.cn/) @ Tsinghua University
由以下团队开发和维护:
- 清华大学 [MADSys 实验室](https://madsys.cs.tsinghua.edu.cn/)
- [Approaching.AI](http://approaching.ai/)
- [9#AISoft](https://github.com/aisoft9)
- Community contributors
- 社区贡献者
We welcome contributions! Please feel free to submit issues and pull requests.
我们欢迎贡献!请随时提交问题和拉取请求。
## 💬 Community & Support
## 💬 社区与支持
- **GitHub Issues**: [Report bugs or request features](https://github.com/kvcache-ai/ktransformers/issues)
- **WeChat Group**: See [archive/WeChatGroup.png](./archive/WeChatGroup.png)
- **GitHub Issues**[报告问题或请求功能](https://github.com/kvcache-ai/ktransformers/issues)
- **微信群**:请参见 [archive/WeChatGroup.png](./archive/WeChatGroup.png)
## 📦 KT original Code
## 📦 KT原仓库
The original integrated KTransformers framework has been archived to the [`archive/`](./archive/) directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.
原始的集成 KTransformers 框架已归档到 [`archive/`](./archive/) 目录以供参考。该项目现在围绕 kt-kernel 源码树中的上述两个能力入口组织文档和维护。
For the original documentation with full quick-start guides and examples, see:
- [archive/README.md](./archive/README.md) (English)
- [archive/README_ZH.md](./archive/README_ZH.md) (中文)
有关原始文档以及完整的快速入门指南和示例,请参见:
- [archive/README.md](./archive/README.md)(英文)
- [archive/README_ZH.md](./archive/README_ZH.md)中文