docs: make Chinese README the default
Book-CI / test (windows-latest) (push) Has been cancelled
Book-CI / test (macos-latest) (push) Has been cancelled
Book-CI / test (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (macos-latest) (push) Has been cancelled
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (windows-latest) (push) Has been cancelled
Book-CI / test (windows-latest) (push) Has been cancelled
Book-CI / test (macos-latest) (push) Has been cancelled
Book-CI / test (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (macos-latest) (push) Has been cancelled
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Deploy / deploy (windows-latest) (push) Has been cancelled
This commit is contained in:
@@ -1,3 +1,9 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/kvcache-ai/ktransformers) · [上游 README](https://github.com/kvcache-ai/ktransformers/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
<div align="center">
|
||||
<p align="center">
|
||||
|
||||
@@ -7,110 +13,99 @@
|
||||
</picture>
|
||||
|
||||
</p>
|
||||
<h3>A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations</h3>
|
||||
<strong><a href="#-overview">🎯 Overview</a> | <a href="#-inference---high-performance-kt-kernel-serving">🚀 Inference</a> | <a href="#-sft---fine-tuning-with-llama-factory">🎓 SFT</a> | <a href="#-citation">🔥 Citation</a> | <a href="https://github.com/kvcache-ai/ktransformers/issues/1921">🚀 Roadmap(2026Q2)</a> </strong>
|
||||
<h3>一个用于体验尖端 LLM 推理/微调优化的灵活框架</h3>
|
||||
<strong><a href="#-概览">🎯 概览</a> | <a href="#-推理---kt-kernel-高性能推理">🚀 推理</a> | <a href="#-sft---llama-factory-微调">🎓 SFT</a> | <a href="#-引用">🔥 引用</a> </strong>
|
||||
</div>
|
||||
|
||||
## 🎯 Overview
|
||||
## 🎯 概览
|
||||
|
||||
KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: [Inference](./kt-kernel/README.md) and [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md).
|
||||
KTransformers 是一个专注于通过 CPU-GPU 异构计算实现大语言模型高效推理和微调的研究项目。目前两个面向用户的能力都来自 kt-kernel 源码目录:[推理](./kt-kernel/README.md) 和 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)。
|
||||
|
||||
## 🔥 Updates
|
||||
* **June 21, 2026**: MiniMax-M3 Day0 Support! ([Tutorial](./doc/en/kt-kernel/MiniMax-M3-Tutorial.md))
|
||||
* **June 17, 2026**: GLM-5.2 Day0 Support! ([Tutorial](./doc/en/kt-kernel/GLM-5.2-Tutorial.md))
|
||||
* **May 6, 2026**: KTransformers at [GOSIM Paris 2026](https://paris2026.gosim.org/zh/schedule/) — "Agentic AI on Edge" track. We'll present KT's inference performance on consumer hardware.
|
||||
* **May 02, 2026**: DeepSeek-V4-Flash Support! ([Tutorial](./doc/en/DeepSeek-V4-Flash.md))
|
||||
* **Apr 30, 2026**: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate [Inference](./kt-kernel/README.md) and [SFT Quick Start](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) entry points.
|
||||
* **Mar 26, 2026**: Support AVX2-only CPU backend for KT-Kernel inference. ([Tutorial](./doc/en/kt-kernel/AVX2-Tutorial.md))
|
||||
* **Feb 13, 2026**: MiniMax-M2.5 Day0 Support! ([Tutorial](./doc/en/MiniMax-M2.5.md))
|
||||
* **Feb 12, 2026**: GLM-5 Day0 Support! ([Tutorial](./doc/en/kt-kernel/GLM-5-Tutorial.md))
|
||||
* **Jan 27, 2026**: Kimi-K2.5 Day0 Support! ([Tutorial](./doc/en/Kimi-K2.5.md)) ([SFT Tutorial](./doc/en/SFT_Installation_Guide_KimiK2.5.md))
|
||||
* **Jan 22, 2026**: Support [CPU-GPU Expert Scheduling](./doc/en/kt-kernel/experts-sched-Tutorial.md), [Native BF16 and FP8 per channel Precision](./doc/en/kt-kernel/Native-Precision-Tutorial.md) and [AutoDL unified fine-tuning and inference](./doc/zh/【云端低价训推】%20KTransformers%2BAutoDL%2BLlamaFactory:随用随租的低成本超大模型「微调%2B推理」一体化流程.pdf)
|
||||
* **Dec 24, 2025**: Support Native MiniMax-M2.1 inference. ([Tutorial](./doc/en/kt-kernel/MiniMax-M2.1-Tutorial.md))
|
||||
* **Dec 22, 2025**: Support RL-DPO fine-tuning with LLaMA-Factory. ([Tutorial](./doc/en/SFT/DPO_tutorial.md))
|
||||
* **Dec 5, 2025**: Support Native Kimi-K2-Thinking inference ([Tutorial](./doc/en/kt-kernel/Kimi-K2-Thinking-Native.md))
|
||||
* **Nov 6, 2025**: Support Kimi-K2-Thinking inference ([Tutorial](./doc/en/Kimi-K2-Thinking.md)) and fine-tune ([Tutorial](./doc/en/SFT_Installation_Guide_KimiK2.md))
|
||||
* **Nov 4, 2025**: KTransformers Fine-Tuning × LLaMA-Factory Integration. ([Tutorial](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md))
|
||||
* **Oct 27, 2025**: Support Ascend NPU. ([Tutorial](./doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md))
|
||||
* **Oct 10, 2025**: Integrating into SGLang. ([Roadmap](https://github.com/sgl-project/sglang/issues/11425), [Blog](https://lmsys.org/blog/2025-10-22-KTransformers/))
|
||||
* **Sept 11, 2025**: Support Qwen3-Next. ([Tutorial](./doc/en/Qwen3-Next.md))
|
||||
* **Sept 05, 2025**: Support Kimi-K2-0905. ([Tutorial](./doc/en/Kimi-K2.md))
|
||||
* **July 26, 2025**: Support SmallThinker and GLM4-MoE. ([Tutorial](./doc/en/SmallThinker_and_Glm4moe.md))
|
||||
* **July 11, 2025**: Support Kimi-K2. ([Tutorial](./doc/en/Kimi-K2.md))
|
||||
* **June 30, 2025**: Support 3-layer (GPU-CPU-Disk) [prefix cache](./doc/en/prefix_cache.md) reuse.
|
||||
* **May 14, 2025**: Support Intel Arc GPU ([Tutorial](./doc/en/xpu.md)).
|
||||
* **Apr 29, 2025**: Support AMX-Int8、 AMX-BF16 and Qwen3MoE ([Tutorial](./doc/en/AMX.md))
|
||||
* **Apr 9, 2025**: Experimental support for LLaMA 4 models ([Tutorial](./doc/en/llama4.md)).
|
||||
* **Apr 2, 2025**: Support Multi-concurrency. ([Tutorial](./doc/en/balance-serve.md)).
|
||||
* **Mar 15, 2025**: Support ROCm on AMD GPU ([Tutorial](./doc/en/ROCm.md)).
|
||||
* **Mar 5, 2025**: Support unsloth 1.58/2.51 bits weights and [IQ1_S/FP8 hybrid](./doc/en/fp8_kernel.md) weights. Support 139K [Longer Context](./doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel) for DeepSeek-V3 and R1 in 24GB VRAM.
|
||||
* **Feb 25, 2025**: Support [FP8 GPU kernel](./doc/en/fp8_kernel.md) for DeepSeek-V3 and R1; [Longer Context](./doc/en/DeepseekR1_V3_tutorial.md#v022-longer-context).
|
||||
* **Feb 15, 2025**: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed (+15%, up to 16 Tokens/s), update [docs](./doc/en/DeepseekR1_V3_tutorial.md) and [online books](https://kvcache-ai.github.io/ktransformers/).
|
||||
* **Feb 10, 2025**: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see [here](./doc/en/DeepseekR1_V3_tutorial.md).
|
||||
* **Aug 28, 2024**: Decrease DeepseekV2's required VRAM from 21G to 11G.
|
||||
* **Aug 15, 2024**: Update detailed [tutorial](doc/en/injection_tutorial.md) for injection and multi-GPU.
|
||||
* **Aug 14, 2024**: Support llamfile as linear backend.
|
||||
* **Aug 12, 2024**: Support multiple GPU; Support new model: mixtral 8\*7B and 8\*22B; Support q2k, q3k, q5k dequant on gpu.
|
||||
* **Aug 9, 2024**: Support windows native.
|
||||
## 🔥 更新
|
||||
|
||||
* **2026 年 4 月 30 日**:KTransformers v0.6.1 更新 kt-kernel 推理和 SFT 文档,提供独立的[推理](./kt-kernel/README.md)和 [SFT Quick Start](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)入口。
|
||||
* **2025 年 12 月 5 日**:支持原生 Kimi-K2-Thinking 推理([教程](./doc/en/kt-kernel/Kimi-K2-Thinking-Native.md))
|
||||
* **2025 年 11 月 6 日**:支持 Kimi-K2-Thinking 推理([教程](./doc/en/Kimi-K2-Thinking.md))和微调([教程](./doc/en/SFT_Installation_Guide_KimiK2.md))
|
||||
* **2025 年 11 月 4 日**:KTransformers 微调 × LLaMA-Factory 集成([教程](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md))
|
||||
* **2025 年 10 月 27 日**:支持昇腾 NPU([教程](./doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md))
|
||||
* **2025 年 10 月 10 日**:集成到 SGLang([路线图](https://github.com/sgl-project/sglang/issues/11425),[博客](https://lmsys.org/blog/2025-10-22-KTransformers/))
|
||||
* **2025 年 9 月 11 日**:支持 Qwen3-Next([教程](./doc/en/Qwen3-Next.md))
|
||||
* **2025 年 9 月 5 日**:支持 Kimi-K2-0905([教程](./doc/en/Kimi-K2.md))
|
||||
* **2025 年 7 月 26 日**:支持 SmallThinker 和 GLM4-MoE([教程](./doc/en/SmallThinker_and_Glm4moe.md))
|
||||
* **2025 年 7 月 11 日**:支持 Kimi-K2([教程](./doc/en/Kimi-K2.md))
|
||||
* **2025 年 6 月 30 日**:支持 3 层(GPU-CPU-磁盘)[前缀缓存](./doc/en/prefix_cache.md)复用
|
||||
* **2025 年 5 月 14 日**:支持 Intel Arc GPU([教程](./doc/en/xpu.md))
|
||||
* **2025 年 4 月 29 日**:支持 AMX-Int8、AMX-BF16 和 Qwen3MoE([教程](./doc/en/AMX.md))
|
||||
* **2025 年 4 月 9 日**:实验性支持 LLaMA 4 模型([教程](./doc/en/llama4.md))
|
||||
* **2025 年 4 月 2 日**:支持多并发([教程](./doc/en/balance-serve.md))
|
||||
* **2025 年 3 月 15 日**:支持 AMD GPU 上的 ROCm([教程](./doc/en/ROCm.md))
|
||||
* **2025 年 3 月 5 日**:支持 unsloth 1.58/2.51 位权重和 [IQ1_S/FP8 混合](./doc/en/fp8_kernel.md)权重。在 24GB VRAM 中支持 DeepSeek-V3 和 R1 的 139K [更长上下文](./doc/en/DeepseekR1_V3_tutorial.md#v022--v023-longer-context--fp8-kernel)
|
||||
* **2025 年 2 月 25 日**:为 DeepSeek-V3 和 R1 支持 [FP8 GPU 内核](./doc/en/fp8_kernel.md);[更长上下文](./doc/en/DeepseekR1_V3_tutorial.md#v022-longer-context)
|
||||
* **2025 年 2 月 15 日**:更长上下文(24GB VRAM 从 4K 到 8K)& 速度稍快(+15%,最高 16 Tokens/s),更新[文档](./doc/en/DeepseekR1_V3_tutorial.md)和[在线手册](https://kvcache-ai.github.io/ktransformers/)
|
||||
* **2025 年 2 月 10 日**:支持 Deepseek-R1 和 V3 在单 GPU(24GB VRAM)/多 GPU 和 382GB DRAM 上运行,速度提升高达 3~28 倍。详细案例展示和复现教程请参见[这里](./doc/en/DeepseekR1_V3_tutorial.md)
|
||||
* **2024 年 8 月 28 日**:将 DeepseekV2 所需的 VRAM 从 21GB 降低到 11GB
|
||||
* **2024 年 8 月 15 日**:更新了关于注入和多 GPU 的详细[教程](doc/en/injection_tutorial.md)
|
||||
* **2024 年 8 月 14 日**:支持 llamfile 作为线性后端
|
||||
* **2024 年 8 月 12 日**:支持多 GPU;支持新模型:mixtral 8\*7B 和 8\*22B;支持 GPU 上的 q2k、q3k、q5k 去量化
|
||||
* **2024 年 8 月 9 日**:支持 Windows 原生环境
|
||||
|
||||
---
|
||||
|
||||
## 📦 Capabilities
|
||||
## 📦 功能入口
|
||||
|
||||
### 🚀 [Inference](./kt-kernel/README.md) - High-Performance kt-kernel Serving
|
||||
### 🚀 [推理](./kt-kernel/README.md) - kt-kernel 高性能推理
|
||||
|
||||
CPU-optimized kernel operations for heterogeneous LLM inference.
|
||||
用于异构 LLM 推理的 CPU 优化内核操作。
|
||||
|
||||
<img width="1049" height="593" alt="image" src="https://github.com/user-attachments/assets/68f423da-3f55-4025-bdc9-9ceaa554f00b" />
|
||||

|
||||
|
||||
**主要特性:**
|
||||
- **AMX/AVX 加速**:Intel AMX 和 AVX512/AVX2 优化的内核,用于 INT4/INT8 量化推理
|
||||
- **MoE 优化**:高效的专家混合推理,具有 NUMA 感知内存管理
|
||||
- **量化支持**:CPU 端 INT4/INT8 量化权重,GPU 端 GPTQ 支持
|
||||
- **易于集成**:为 SGLang 和其他框架提供简洁的 Python API
|
||||
|
||||
**Key Features:**
|
||||
- **AMX/AVX Acceleration**: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
|
||||
- **MoE Optimization**: Efficient Mixture-of-Experts inference with NUMA-aware memory management
|
||||
- **Quantization Support**: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
|
||||
- **Easy Integration**: Clean Python API for SGLang and other frameworks
|
||||
|
||||
**Quick Start:**
|
||||
**快速开始:**
|
||||
```bash
|
||||
cd kt-kernel
|
||||
pip install .
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
**使用场景:**
|
||||
|
||||
- CPU-GPU hybrid inference for large MoE models
|
||||
- Integration with SGLang for production serving
|
||||
- Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)
|
||||
- 大型 MoE 模型的 CPU-GPU 混合推理
|
||||
- 与 SGLang 集成用于生产服务
|
||||
- 异构专家放置(热专家在 GPU 上,冷专家在 CPU 上)
|
||||
|
||||
**Performance Examples:**
|
||||
| Model | Hardware Configuration | Total Throughput | Output Throughput |
|
||||
**性能示例:**
|
||||
| 模型 | 硬件配置 | 总吞吐量 | 输出吞吐量 |
|
||||
|-------|------------------------|------------------|-------------------|
|
||||
| DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens/s | 87.58 tokens/s (8-way concurrency) |
|
||||
| DeepSeek-R1-0528 (FP8) | 8×L20 GPU + Xeon Gold 6454S | 227.85 tokens/s | 87.58 tokens/s(8 路并发)|
|
||||
|
||||
👉 **[Full Documentation →](./kt-kernel/README.md)**
|
||||
👉 **[完整文档 →](./kt-kernel/README.md)**
|
||||
|
||||
---
|
||||
|
||||
### 🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - Fine-Tuning with LLaMA-Factory
|
||||
### 🎓 [SFT](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md) - LLaMA-Factory 微调
|
||||
|
||||
KTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.
|
||||
KTransformers × LLaMA-Factory 集成,面向超大 MoE 模型微调。
|
||||
|
||||

|
||||

|
||||
|
||||
**Key Features:**
|
||||
- **Multi-Backend Support**: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
|
||||
- **Ultra-Large MoE Support**: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
|
||||
- **Faster than ZeRO-Offload**: 6-12x training speedup in benchmarked MoE SFT workloads
|
||||
- **Lower CPU Memory**: About half the CPU memory of the previous KT SFT path in the benchmarked setup
|
||||
- **LLaMA-Factory Integration**: Seamless integration with popular fine-tuning framework
|
||||
**主要特性:**
|
||||
- **多后端支持**: CPU/GPU 混合微调,支持 INT8/INT4 量化
|
||||
- **超大 MoE 支持**: 在有限 GPU 内存下微调 DeepSeek-V3/R1 等模型
|
||||
- **相对 ZeRO-Offload 加速**: 在基准 MoE SFT 任务中训练速度提升 6-12 倍
|
||||
- **降低 CPU 内存**: 相比上一版 KT SFT 路径,基准配置下 CPU 内存约降至 1/2
|
||||
- **LLaMA-Factory 集成**: 与流行微调框架无缝集成
|
||||
|
||||
| Model | GPU Memory | Training Speed | Hardware |
|
||||
| 模型 | GPU 内存 | 训练速度 | 硬件 |
|
||||
|-------|------------|----------------|----------|
|
||||
| DeepSeek-V3 | ~80GB total | 3.7 it/s | 4x RTX 4090 |
|
||||
| DeepSeek-R1 | ~80GB total | 3.7 it/s | 4x RTX 4090 |
|
||||
| Qwen3-30B-A3B | ~24GB total | 8+ it/s | 1x RTX 4090 |
|
||||
| DeepSeek-V3 | ~80GB 总计 | 3.7 it/s | 4x RTX 4090 |
|
||||
| DeepSeek-R1 | ~80GB 总计 | 3.7 it/s | 4x RTX 4090 |
|
||||
| Qwen3-30B-A3B | ~24GB 总计 | 8+ it/s | 1x RTX 4090 |
|
||||
|
||||
**Quick Start:**
|
||||
**快速开始:**
|
||||
```bash
|
||||
cd /path/to/LLaMA-Factory
|
||||
pip install -e .
|
||||
@@ -122,13 +117,13 @@ CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
|
||||
```
|
||||
|
||||
👉 **[Quick Start →](./doc/en/SFT/KTransformers-Fine-Tuning_Quick-Start.md)**
|
||||
👉 **[Full Documentation →](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md)**
|
||||
👉 **[完整文档 →](./doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md)**
|
||||
|
||||
---
|
||||
|
||||
## 🔥 Citation
|
||||
## 🔥 引用
|
||||
|
||||
If you use KTransformers in your research, please cite our paper:
|
||||
如果您在研究中使用了 KTransformers,请引用我们的论文:
|
||||
|
||||
```bibtex
|
||||
@inproceedings{10.1145/3731569.3764843,
|
||||
@@ -139,25 +134,24 @@ If you use KTransformers in your research, please cite our paper:
|
||||
}
|
||||
```
|
||||
|
||||
## 👥 Contributors & Team
|
||||
## 👥 贡献者与团队
|
||||
|
||||
Developed and maintained by:
|
||||
- [MADSys Lab](https://madsys.cs.tsinghua.edu.cn/) @ Tsinghua University
|
||||
由以下团队开发和维护:
|
||||
- 清华大学 [MADSys 实验室](https://madsys.cs.tsinghua.edu.cn/)
|
||||
- [Approaching.AI](http://approaching.ai/)
|
||||
- [9#AISoft](https://github.com/aisoft9)
|
||||
- Community contributors
|
||||
- 社区贡献者
|
||||
|
||||
We welcome contributions! Please feel free to submit issues and pull requests.
|
||||
我们欢迎贡献!请随时提交问题和拉取请求。
|
||||
|
||||
## 💬 Community & Support
|
||||
## 💬 社区与支持
|
||||
|
||||
- **GitHub Issues**: [Report bugs or request features](https://github.com/kvcache-ai/ktransformers/issues)
|
||||
- **WeChat Group**: See [archive/WeChatGroup.png](./archive/WeChatGroup.png)
|
||||
- **GitHub Issues**:[报告问题或请求功能](https://github.com/kvcache-ai/ktransformers/issues)
|
||||
- **微信群**:请参见 [archive/WeChatGroup.png](./archive/WeChatGroup.png)
|
||||
|
||||
## 📦 KT original Code
|
||||
## 📦 KT原仓库
|
||||
|
||||
The original integrated KTransformers framework has been archived to the [`archive/`](./archive/) directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.
|
||||
原始的集成 KTransformers 框架已归档到 [`archive/`](./archive/) 目录以供参考。该项目现在围绕 kt-kernel 源码树中的上述两个能力入口组织文档和维护。
|
||||
|
||||
For the original documentation with full quick-start guides and examples, see:
|
||||
- [archive/README.md](./archive/README.md) (English)
|
||||
- [archive/README_ZH.md](./archive/README_ZH.md) (中文)
|
||||
有关原始文档以及完整的快速入门指南和示例,请参见:
|
||||
- [archive/README.md](./archive/README.md)(英文)
|
||||
- [archive/README_ZH.md](./archive/README_ZH.md)(中文)
|
||||
|
||||
Reference in New Issue
Block a user