- | Model |
- Parameters |
+ 模型 |
+ 参数量 |
CPU |
Kernel |
@@ -156,34 +162,34 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
-## Installation
+## 安装
-### Requirements
+### 环境要求
- python>=3.9
- cmake>=3.22
- clang>=18
- - For Windows users, install [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/). In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):
+ - 对于 Windows 用户，请安装 [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/).。在安装程序中，请至少勾选以下选项（这也会自动安装 CMake 等所需的其他工具）：
- Desktop-development with C++
- C++-CMake Tools for Windows
- Git for Windows
- C++-Clang Compiler for Windows
- MS-Build Support for LLVM-Toolset (clang)
- - For Debian/Ubuntu users, you can download with [Automatic installation script](https://apt.llvm.org/)
+ - 对于 Debian/Ubuntu 用户，可通过[自动安装脚本](https://apt.llvm.org/) 下载：
`bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"`
-- conda (highly recommend)
+- conda（强烈推荐）
-### Build from source
+### 从源码构建
> [!IMPORTANT]
-> If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands. Please refer to the FAQs below if you see any issues.
+> 如果你使用的是 Windows，请务必始终使用 VS2022 的 Developer Command Prompt / PowerShell 执行以下命令。如遇到问题，请参阅下方的常见问题解答（FAQs）。
-1. Clone the repo
+1. 克隆仓库
```bash
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
```
-2. Install the dependencies
+2. 安装依赖
```bash
# (Recommended) Create a new conda environment
conda create -n bitnet-cpp python=3.9
@@ -191,7 +197,7 @@ conda activate bitnet-cpp
pip install -r requirements.txt
```
-3. Build the project
+3. 构建项目
```bash
# Manually download the model and run with local path
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
@@ -201,24 +207,25 @@ python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
usage: setup_env.py [-h] [--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}] [--model-dir MODEL_DIR] [--log-dir LOG_DIR] [--quant-type {i2_s,tl1}] [--quant-embd]
[--use-pretuned]
+
-Setup the environment for running inference
+配置用于运行推理的环境
optional arguments:
- -h, --help show this help message and exit
+ -h, --help 显示此帮助信息并退出
--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}, -hr {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}
- Model used for inference
+ 用于推理的模型
--model-dir MODEL_DIR, -md MODEL_DIR
- Directory to save/load the model
+ 保存/加载模型的目录
--log-dir LOG_DIR, -ld LOG_DIR
- Directory to save the logging info
+ 保存日志信息的目录
--quant-type {i2_s,tl1}, -q {i2_s,tl1}
- Quantization type
- --quant-embd Quantize the embeddings to f16
- --use-pretuned, -p Use the pretuned kernel parameters
+ 量化类型
+ --quant-embd 将嵌入层量化到 f16
+ --use-pretuned, -p 使用预调优的内核参数
-## Usage
-### Basic usage
+## 用法
+### 基本用法
```bash
# Run inference with the quantized model
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
@@ -229,25 +236,25 @@ usage: run_inference.py [-h] [-m MODEL] [-n N_PREDICT] -p PROMPT [-t THREADS] [-
Run inference
optional arguments:
- -h, --help show this help message and exit
+ -h, --help 显示此帮助信息并退出
-m MODEL, --model MODEL
- Path to model file
+ 模型文件路径
-n N_PREDICT, --n-predict N_PREDICT
- Number of tokens to predict when generating text
+ 生成文本时要预测的 token 数量
-p PROMPT, --prompt PROMPT
- Prompt to generate text from
+ 用于生成文本的提示词
-t THREADS, --threads THREADS
- Number of threads to use
+ 使用的线程数
-c CTX_SIZE, --ctx-size CTX_SIZE
- Size of the prompt context
+ 提示词上下文大小
-temp TEMPERATURE, --temperature TEMPERATURE
- Temperature, a hyperparameter that controls the randomness of the generated text
- -cnv, --conversation Whether to enable chat mode or not (for instruct models.)
- (When this option is turned on, the prompt specified by -p will be used as the system prompt.)
+ Temperature（温度），控制生成文本随机性的超参数
+ -cnv, --conversation 是否启用聊天模式（适用于 instruct 模型。）
+ （启用此选项时，-p 指定的提示词将作为系统提示词使用。）
-### Benchmark
-We provide scripts to run the inference benchmark providing a model.
+### 基准测试
+我们提供了脚本，可在给定模型的情况下运行推理基准测试。
```
usage: e2e_benchmark.py -m MODEL [-n N_TOKEN] [-p N_PROMPT] [-t THREADS]
@@ -269,23 +276,23 @@ optional arguments:
Number of threads to use.
```
-Here's a brief explanation of each argument:
+以下简要说明各参数：
-- `-m`, `--model`: The path to the model file. This is a required argument that must be provided when running the script.
-- `-n`, `--n-token`: The number of tokens to generate during the inference. It is an optional argument with a default value of 128.
-- `-p`, `--n-prompt`: The number of prompt tokens to use for generating text. This is an optional argument with a default value of 512.
-- `-t`, `--threads`: The number of threads to use for running the inference. It is an optional argument with a default value of 2.
-- `-h`, `--help`: Show the help message and exit. Use this argument to display usage information.
+- `-m`、`--model`：模型文件路径。运行脚本时必须提供此必填参数。
+- `-n`、`--n-token`：推理过程中要生成的 token 数量。可选参数，默认值为 128。
+- `-p`、`--n-prompt`：用于生成文本的提示词 token 数量。可选参数，默认值为 512。
+- `-t`、`--threads`：运行推理时使用的线程数。可选参数，默认值为 2。
+- `-h`、`--help`：显示帮助信息并退出。使用此参数可查看用法说明。
-For example:
+例如：
```sh
python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4
```
-This command would run the inference benchmark using the model located at `/path/to/model`, generating 200 tokens from a 256 token prompt, utilizing 4 threads.
+此命令将使用位于 `/path/to/model` 的模型运行推理基准测试，从 256 个 token 的提示词生成 200 个 token，并使用 4 个线程。
-For the model layout that do not supported by any public model, we provide scripts to generate a dummy model with the given model layout, and run the benchmark on your machine:
+对于没有任何公开模型支持的模型布局，我们提供了脚本，可根据给定模型布局生成虚拟模型，并在你的机器上运行基准测试：
```bash
python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M
@@ -294,7 +301,7 @@ python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128
```
-### Convert from `.safetensors` Checkpoints
+### 从 `.safetensors` 检查点转换
```sh
# Prepare the .safetensors model file
@@ -304,36 +311,36 @@ huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./models/
python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16
```
-### FAQ (Frequently Asked Questions)📌
+### 常见问题（FAQ）📌
-#### Q1: The build dies with errors building llama.cpp due to issues with std::chrono in log.cpp?
+#### Q1：构建时因在 log.cpp 中 std::chrono 相关问题导致编译 llama.cpp 失败？
-**A:**
-This is an issue introduced in recent version of llama.cpp. Please refer to this [commit](https://github.com/tinglou/llama.cpp/commit/4e3db1e3d78cc1bcd22bcb3af54bd2a4628dd323) in the [discussion](https://github.com/abetlen/llama-cpp-python/issues/1942) to fix this issue.
+**A：**
+这是 llama.cpp 近期版本中引入的问题。请参阅此 [commit](https://github.com/tinglou/llama.cpp/commit/4e3db1e3d78cc1bcd22bcb3af54bd2a4628dd323) in the [discussion](https://github.com/abetlen/llama-cpp-python/issues/1942) 以修复此问题。
-#### Q2: How to build with clang in conda environment on windows?
+#### Q2：如何在 Windows 上的 conda 环境中使用 clang 构建？
-**A:**
-Before building the project, verify your clang installation and access to Visual Studio tools by running:
+**A：**
+在构建项目之前，请运行以下命令验证 clang 安装及 Visual Studio 工具是否可用：
```
clang -v
```
-This command checks that you are using the correct version of clang and that the Visual Studio tools are available. If you see an error message such as:
+此命令会检查你是否使用了正确版本的 clang，以及 Visual Studio 工具是否可用。如果你看到如下错误信息：
```
'clang' is not recognized as an internal or external command, operable program or batch file.
```
-It indicates that your command line window is not properly initialized for Visual Studio tools.
+这表明你的命令行窗口未正确初始化 Visual Studio 工具。
-• If you are using Command Prompt, run:
+• 如果你使用命令提示符（Command Prompt），请运行：
```
"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64
```
-• If you are using Windows PowerShell, run the following commands:
+• 如果你使用 Windows PowerShell，请运行以下命令：
```
Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll" Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"
```
-These steps will initialize your environment and allow you to use the correct Visual Studio tools.
+这些步骤将初始化你的环境，使你能够使用正确的 Visual Studio 工具。
](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)
+[
-## Demo
+## 演示
-A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:
+在 Apple M2 上运行 BitNet b1.58 3B 模型的 bitnet.cpp 演示：
https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
-## What's New:
+## 最新动态：
- 01/15/2026 [BitNet CPU Inference Optimization](https://github.com/microsoft/BitNet/blob/main/src/README.md) 
- 05/20/2025 [BitNet Official GPU inference kernel](https://github.com/microsoft/BitNet/blob/main/gpu/README.md)
- 04/14/2025 [BitNet Official 2B Parameter Model on Hugging Face](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)
@@ -33,15 +39,15 @@ https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
- 02/27/2024 [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764)
- 10/17/2023 [BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453)
-## Acknowledgements
+## 致谢
-This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp) framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered in [T-MAC](https://github.com/microsoft/T-MAC/). For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.
-## Official Models
+本项目基于 [llama.cpp](https://github.com/ggerganov/llama.cpp) 框架。我们感谢所有作者对开源社区的贡献。此外，bitnet.cpp 的 kernel 构建于 [T-MAC](https://github.com/microsoft/T-MAC/). 开创的查找表（Lookup Table）方法之上。对于三值模型之外的一般低位 LLM 推理，我们建议使用 T-MAC。
+## 官方模型