From 195bbdb3b7dde01686f85a6ee235f2a1361b5d28 Mon Sep 17 00:00:00 2001 From: wehub-resource-sync Date: Mon, 13 Jul 2026 10:11:56 +0000 Subject: [PATCH] docs: make Chinese README the default --- README.md | 988 ++++++++++++++++++------------------------------------ 1 file changed, 317 insertions(+), 671 deletions(-) diff --git a/README.md b/README.md index 7856952..1be2fa7 100644 --- a/README.md +++ b/README.md @@ -1,108 +1,67 @@ + +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/antirez/ds4) · [上游 README](https://github.com/antirez/ds4/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 + # DwarfStar -**DwarfStar** is a small native inference engine optimized first for -**DeepSeek V4 Flash**, with support for **DeepSeek V4 PRO** on very high-memory -machines. It is -intentionally narrow: not a generic GGUF runner, not a wrapper around another -runtime: it is completely self-contained. Other than running the model in a -correct and fast way, the project goal is to provide DeepSeek specific loading, -prompt rendering, tool calling, KV state handling (RAM and on-disk), server -API and integrated coding agent, all ready to work with coding agents or with -the provided CLI interface. There are also tools for GGUF and imatrix generation, -and for quality and speed testing. +**DwarfStar** 是一款小型原生推理引擎，优先针对 **DeepSeek V4 Flash** 优化，并在超高内存机器上支持 **DeepSeek V4 PRO**。它的定位刻意收窄：不是通用的 GGUF 运行器，也不是对另一套运行时的封装——完全自包含。除正确、快速地运行模型外，项目目标还包括提供 DeepSeek 专属的加载、提示词渲染、工具调用、KV 状态处理（内存与磁盘）、服务端 API 以及集成的编程智能体，开箱即可与编程智能体或自带的 CLI 界面配合使用。此外还提供 GGUF 与 imatrix 生成工具，以及质量与速度测试工具。 -We support the following backends: -* **Metal** is our primary target. Starting from MacBooks with 96GB of RAM (or less, using SSD streaming). -* **NVIDIA CUDA / DGX Spark**, CUDA with special care for the DGX Spark. -* **Strix Halo (ROCm)**, systems like the Framework Desktop and other systems based on the same GPU and unified RAM design. +我们支持以下后端： +* **Metal** 是主要目标平台。从配备 96GB RAM 的 MacBook 起步（或更少内存时通过 SSD 流式加载）。 +* **NVIDIA CUDA / DGX Spark**，CUDA 对 DGX Spark 做了特别优化。 +* **Strix Halo (ROCm)**，如 Framework Desktop 及采用相同 GPU 与统一内存设计的其他系统。 -This project would not exist without **llama.cpp and GGML**, make sure to read -the acknowledgements section, a big thank you to Georgi Gerganov and all the -other contributors. +没有 **llama.cpp 与 GGML**，本项目就不会存在；请务必阅读致谢章节，衷心感谢 Georgi Gerganov 及所有其他贡献者。 -**Note that DeepSeek v4** is not our only target. Right now Flash and PRO are the -perfect fit because of capabilities, size, KV cache efficiency. If tomorrow a -better open weight model is released for the 128GB size, we could switch, the same -for other important size classes like 512GB of RAM. The project is stictly -opportunistic depending on what open weight models exist in a given moment. -If a new model will be supported, the old one may be removed completely and -no longer supported, unless there is some kind of overlap of abilities. +**请注意，DeepSeek v4** 并非我们唯一的目标。目前 Flash 与 PRO 因能力、体量与 KV cache 效率而最为契合。若日后有更适合 128GB 档位的更优开放权重模型发布，我们可能会切换；其他重要内存档位（如 512GB RAM）同理。项目严格随当下可用的开放权重模型而调整，机会主义取向明显。若支持新模型，旧模型可能被完全移除且不再维护，除非二者能力存在某种重叠。 -## Motivations +## 动机 -* Very capable open weight models finally exist. DeepSeek v4 Flash feels quasi-frontier. The PRO is even better. Both resist 2 bit quantization very well. -* Very capable computers like MacBooks, the DGX Spark now exist. -* DeepSeek v4 kv cache design makes it pratical to run very big contexts. Other vendors are using this approach. -* This few hundred billions models are strictly better than smaller (even if dense) models, regardless of what benchmarks say. +* 终于出现了能力很强的开放权重模型。DeepSeek v4 Flash 已接近前沿水准，PRO 更强。二者对 2 bit 量化都表现出很好的鲁棒性。 +* MacBook、DGX Spark 等高性能计算机现已普及。 +* DeepSeek v4 的 KV cache 设计使运行超长上下文变得切实可行；其他厂商也在采用类似思路。 +* 这类数千亿参数规模的模型，严格优于更小（即便为稠密）的模型，与基准测试结论无关。 -That said, a few important things about this project: +话虽如此，关于本项目还有几点重要说明： -* The local inference landscape contains many excellent projects, but new models are released continuously, and the attention immediately gets captured by the next model to implement. This project takes a deliberately narrow bet: one model at a time, official-vector validation (logits obtained with the official implementation), long-context tests, and enough agent integration to know if it really works. The exact model may change as the landscape evolves, but the constraint remains: local inference credible on high end personal machines or Mac Studios, starting from 96/128GB of memory. -* This software is developed with **strong assistance from GPT 5.5** and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without `llama.cpp` and GGML, largely written by hand. -* This implementation is based on the idea that compressed KV caches like the one of DeepSeek v4 and the fast SSD disks of modern MacBooks should change our idea that KV cache belongs to RAM. **The KV cache is actually a first-class disk citizen**. Fast SSD disks also changed the inference game from the point of view of "model needs to fit RAM": while having more RAM the the model size is still preferred, SSD streaming allows to turn the available amount of RAM from a hard cutoff (can I run this model or not?) to continuous spectrum of speed levels. -* Our vision is that local inference should be a set of three things working well together, out of the box: A) inference engine with HTTP API + B) GGUF specially crafted to run well under a given engine and given assumptions + C) testing and validation with coding agents implementations. D) Purpose built agents for specific models and execution environments. DwarfStar only runs with the GGUF files provided. It gets tested against officially obtained logits at different context sizes. This project exists because we wanted to make one local model feel finished end to end, not just runnable. However this is beta quality code, so probably we are not still there, especially since recently we introduced large new features: distributed inference, SSD streaming, and other minor improvements. -* The optimized graph path targets **Metal on macOS** and **CUDA on Linux**. The CPU path is only for correctness checks and model/tokenizer diagnostics. For CPU-only Linux builds, use `make cpu`; it builds the normal `./ds4` and `./ds4-server` binaries without CUDA or Metal. On macOS, **warning: current macOS versions have a bug in the virtual memory implementation that will crash the kernel** if you try to run the CPU code. Remember? Software sucks. It was not possible to fix the CPU inference to avoid crashing, since each time you have to restart the computer, which is not funny. Help us, if you have the guts. +* 本地推理领域已有许多优秀项目，但新模型持续发布，注意力立刻被下一个待实现的模型吸走。本项目刻意押注窄路线：一次只聚焦一个模型、官方向量校验（logits 与官方实现一致）、长上下文测试，以及足以判断“是否真正可用”的智能体集成。具体模型会随生态演变而更换，但约束不变：在高端个人机或 Mac Studio 上实现可信的本地推理，起步内存为 96/128GB。 +* 本软件在 **GPT 5.5** 强力辅助下开发，由人类主导思路、测试与调试。我们公开说明这一点，因为它塑造了项目的构建方式。若您不接受 AI 参与开发的代码，本软件不适合您。下文致谢同样重要：没有 `llama.cpp` 与 GGML，本项目不会存在，其中大量代码为手写。 +* 本实现基于一个观点：像 DeepSeek v4 这类压缩 KV cache，加上现代 MacBook 的高速 SSD，应改变“KV cache 只属于 RAM”的固有认知。**KV cache 实际上是磁盘上的一等公民**。高速 SSD 也从“模型必须装进 RAM”的角度改变了推理格局：更多 RAM 仍优于模型体积固然理想，但 SSD 流式加载能把可用 RAM 从硬性门槛（能否跑起该模型？）变为连续的速度谱系。 +* 我们的愿景是：本地推理应由三件事开箱即用、协同良好：A) 带 HTTP API 的推理引擎 + B) 针对特定引擎与假设精心打造的 GGUF + C) 借助编程智能体实现进行测试与校验 + D) 面向特定模型与运行环境的专用智能体。DwarfStar 仅使用本项目提供的 GGUF 文件运行，并在不同上下文长度下与官方 logits 对比测试。本项目存在，是因为我们希望让一款本地模型在端到端体验上“完成度”十足，而不仅是“能跑”。不过代码尚属 beta 质量，恐怕尚未完全达到，尤其近期我们引入了分布式推理、SSD 流式加载等大型新特性及其他若干改进。 +* 优化图路径面向 **macOS 上的 Metal** 与 **Linux 上的 CUDA**。CPU 路径仅用于正确性检查与模型/分词器诊断。纯 CPU 的 Linux 构建请使用 `make cpu`；它会构建常规的 `./ds4` 与 `./ds4-server` 二进制，不含 CUDA 或 Metal。在 macOS 上，**警告：当前 macOS 版本的虚拟内存实现存在缺陷，运行 CPU 代码会导致内核崩溃**。还记得吗？软件真糟。无法修复 CPU 推理以避免崩溃——每次都得重启电脑，并不好笑。若您有勇气，请帮我们。 -## Acknowledgements to llama.cpp and GGML +## 对 llama.cpp 与 GGML 的致谢 -`ds4.c` does not link against GGML, but it **exists thanks to the path opened by the -llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won -engineering knowledge developed there**. -We are thankful and indebted to [`llama.cpp`](https://github.com/ggml-org/llama.cpp) -and its contributors. Their implementation, kernels, tests, and design choices were -an essential reference while building this DeepSeek V4 specific inference path. -Some source-level pieces are retained or adapted here under the MIT license: GGUF -quant layouts and tables, CPU quant/dot logic, and certain kernels. For this -reason, and because we are genuinely grateful, we keep the GGML authors copyright -notice in our `LICENSE` file. +`ds4.c` 并不链接 GGML，但**得益于 llama.cpp 项目开辟的道路，以及其中开发的内核、量化格式、GGUF 生态与来之不易的工程经验**才得以存在。 +我们感谢并亏欠 [`llama.cpp`](https://github.com/ggml-org/llama.cpp) +) 及其贡献者。他们在实现、内核、测试与设计选择上的工作，是构建这条 DeepSeek V4 专用推理路径时的重要参考。部分源码级片段在本项目中保留或改编，遵循 MIT 许可证：GGUF 量化布局与表、CPU 量化/点积逻辑，以及部分内核。因此，且出于真诚的感激，我们在 `LICENSE` 文件中保留 GGML 作者的版权声明。 -## Status +## 状态 -The code and GGUF files are to be considered of **beta quality** because -inference and model serving is a complicated matter and all this exists -only for a few days. It will take months to reach a more stable form. -However, we try to keep the project in a usable state, and we are making -progress. If you have issues, make sure to use `--trace` to log the -sessions, and open issues including the full trace. +代码与 GGUF 文件应视为 **beta 质量**：推理与模型服务本身很复杂，而这一切问世仅数日。达到更稳定形态尚需数月。不过我们尽力保持项目可用，并在持续进步。若遇到问题，请使用 `--trace` 记录会话，并在提 issue 时附上完整 trace。 -The `ds4-agent` is alpha quality, the project was later added. +`ds4-agent` 为 alpha 质量，是项目后期加入的。 -## More Documentation +## 更多文档 -If you are looking for very specific things, we have other -sub-README files. Otherwise for normal usage keep reading the -next sections. +若您在查找非常具体的内容，可参阅其他子 README 文件。一般使用请继续阅读下文各节。 -- [CONTRIBUTING.md](CONTRIBUTING.md): correctness and speed regression testing - guide for contributors. **Read this before sending a pull request**. -- [gguf-tools/README.md](gguf-tools/README.md): offline GGUF generation, - imatrix collection, quantization tooling, and quality checks. -- [gguf-tools/imatrix/README.md](gguf-tools/imatrix/README.md): how the - routed-MoE imatrix is collected and used. -- [gguf-tools/imatrix/dataset/README.md](gguf-tools/imatrix/dataset/README.md): - how the calibration prompt corpus is generated. -- [gguf-tools/quality-testing/README.md](gguf-tools/quality-testing/README.md): - how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations. -- [dir-steering/README.md](dir-steering/README.md): directional steering data, - vector generation, and usage. -- [speed-bench/README.md](speed-bench/README.md): benchmark commands, charts, - and CSV generation. -- [tests/test-vectors/README.md](tests/test-vectors/README.md): official - continuation vectors used for regression checks. +- [CONTRIBUTING.md](CONTRIBUTING.md)：面向贡献者的正确性与速度回归测试指南。**提交 pull request 前请先阅读**。 +- [gguf-tools/README.md](gguf-tools/README.md)：离线 GGUF 生成、imatrix 采集、量化工具与质量检查。 +- [gguf-tools/imatrix/README.md](gguf-tools/imatrix/README.md)：routed-MoE imatrix 的采集与使用方式。 +- [gguf-tools/imatrix/dataset/README.md](gguf-tools/imatrix/dataset/README.md)：校准提示词语料库的生成方式。 +- [gguf-tools/quality-testing/README.md](gguf-tools/quality-testing/README.md)：本地 GGUF 相对官方 DeepSeek V4 Flash/PRO 续写结果的评分方式。 +- [dir-steering/README.md](dir-steering/README.md)：方向引导（directional steering）数据、向量生成与使用。 +- [speed-bench/README.md](speed-bench/README.md)：基准测试命令、图表与 CSV 生成。 +- [tests/test-vectors/README.md](tests/test-vectors/README.md)：用于回归检查的官方续写向量。 -## Model Weights +## 模型权重 -This implementation only works with the DeepSeek V4 Flash and PRO GGUFs published for -this project. It is not a general GGUF loader, and arbitrary DeepSeek/GGUF files -will not have the tensor layout, quantization mix, metadata, or optional MTP -state expected by the engine. The 2 bit quantizations provided here are not -a joke: they behave well, work under coding agents, call tools in a reliable way. -The 2 bit quants use a very asymmetrical quantization: only the routed MoE -experts are quantized, up/gate at `IQ2_XXS`, down at `Q2_K`. They are the -majority of all the model space: the other components (shared experts, -projections, routing) are left untouched to guarantee quality. +本实现仅适用于本项目发布的 DeepSeek V4 Flash 与 PRO GGUF。它不是通用 GGUF 加载器；任意 DeepSeek/GGUF 文件未必具备引擎所期望的张量布局、量化组合、元数据或可选 MTP 状态。此处提供的 2 bit 量化并非儿戏：表现良好，可在编程智能体下工作，并能可靠地调用工具。2 bit 量化采用高度非对称策略：仅对 routed MoE 专家量化，up/gate 为 `IQ2_XXS`，down 为 `Q2_K`。它们占模型空间的主体；其余组件（共享专家、投影、路由）保持原样，以保证质量。 -Download one main model. **Prefer the imatrix versions.** +下载一个主模型。**优先选择 imatrix 版本。** ```sh ./download_model.sh q2-imatrix # 96/128 GB RAM machines, imatrix-tuned q2 @@ -111,35 +70,35 @@ Download one main model. **Prefer the imatrix versions.** ./download_model.sh pro-q2-imatrix # 512 GB RAM machines, PRO q2 imatrix quant ``` -For the full PRO Q4 distributed run, download one half on each machine: +若要完整运行分布式 PRO Q4，请在每台机器上下载一半： ```sh ./download_model.sh pro-q4-layers00-30 # first half of PRO Q4 split ./download_model.sh pro-q4-layers31-output # second half of PRO Q4 split ``` -The script downloads from `https://huggingface.co/antirez/deepseek-v4-gguf`, -stores files under `./gguf/`, resumes partial downloads with `curl -C -`, and -updates `./ds4flash.gguf` to point at the selected main model. -The `pro-q4-layers00-30`, `pro-q4-layers31-output`, and `pro-q4-split` targets -download distributed PRO Q4 pieces and do not update `./ds4flash.gguf`. -Authentication is optional for public downloads, but `--token TOKEN`, -`HF_TOKEN`, or the local Hugging Face token cache are used when present. +该脚本从 `https://huggingface.co/antirez/deepseek-v4-gguf` 下载， +将文件存储在 `./gguf/` 下，使用 `curl -C -` 恢复部分下载，并 +更新 `./ds4flash.gguf` 以指向所选主模型。 +`pro-q4-layers00-30`、`pro-q4-layers31-output` 和 `pro-q4-split` 目标 +会下载分布式 PRO Q4 分片，且不会更新 `./ds4flash.gguf`。 +公开下载无需认证，但在可用时会使用 `--token TOKEN`、 +`HF_TOKEN` 或本地 Hugging Face token 缓存。 -If you want to regenerate GGUF files or collect a new imatrix, see -[gguf-tools/README.md](gguf-tools/README.md). Those tools are meant for offline -model-building work and can take a long time on the full DeepSeek V4 Flash -weights. Flash GGUF generation is supported by the local tools. PRO GGUF -production currently still depends on the external `llama.cpp`-based workflow; -native tooling can be added later. +若要重新生成 GGUF 文件或收集新的 imatrix，请参阅 +[gguf-tools/README.md](gguf-tools/README.md)。这些工具用于离线 +模型构建工作，在完整 DeepSeek V4 Flash +权重上可能耗时很长。本地工具支持 Flash GGUF 生成。PRO GGUF +制作目前仍依赖基于 `llama.cpp` 的外部工作流； +后续可添加原生工具支持。 -`./download_model.sh mtp` fetches the optional speculative decoding support -GGUF for Flash. It can be used with q2-imatrix, q2-q4-imatrix, and q4-imatrix, -but must be enabled explicitly with `--mtp`. The current MTP/speculative -decoding path is still experimental: it is correctness-gated and currently -provides at most a slight speedup, not a meaningful generation-speed win. +`./download_model.sh mtp` 会拉取 Flash 可选的推测解码（speculative decoding）支持 +GGUF。可与 q2-imatrix、q2-q4-imatrix 和 q4-imatrix 搭配使用， +但必须通过 `--mtp` 显式启用。当前的 MTP/推测 +解码路径仍处于实验阶段：需通过正确性门控，且目前 +最多只能带来轻微加速，无法显著提升生成速度。 -Then build: +然后构建： ```sh make # macOS Metal @@ -148,18 +107,18 @@ make cuda-generic # Linux CUDA, other local CUDA GPUs make cpu # CPU-only diagnostics build ``` -`./ds4flash.gguf` is the default model path used by both binaries. Pass `-m` to -select another supported GGUF from `./gguf/`. Run `./ds4 --help` and -`./ds4-server --help` for the full flag list. +`./ds4flash.gguf` 是两个二进制文件使用的默认模型路径。传入 `-m` 以 +从 `./gguf/` 中选择另一个受支持的 GGUF。运行 `./ds4 --help` 和 +`./ds4-server --help` 可查看完整参数列表。 -## Speed +## 速度 -These are single-run Metal CLI numbers with `--ctx 32768`, `--nothink`, greedy -decoding, and `-n 256`. The short prompt is a normal small Italian story -prompt. The long prompts exercise chunked prefill plus long-context decode. -Q4 requires the larger-memory machine class, so M3 Max Q4 numbers are `N/A`. +以下为单次运行的 Metal CLI 数据，使用 `--ctx 32768`、`--nothink`、贪婪 +解码和 `-n 256`。短提示词为普通的小型意大利语故事 +提示词。长提示词用于测试分块预填充（chunked prefill）加长上下文解码。 +Q4 需要更大内存的机器档次，因此 M3 Max 的 Q4 数据为 `N/A`。 -| Machine | Quant | Prompt | Prefill | Generation | +| 机器 | 量化 | 提示词 | 预填充 | 生成 | | --- | ---: | ---: | ---: | ---: | | MacBook Pro M3 Max, 128 GB | q2 | short | 58.52 t/s | 26.68 t/s | | MacBook Pro M3 Max, 128 GB | q2 | 11709 tokens | 250.11 t/s | 21.47 t/s | @@ -177,47 +136,46 @@ Q4 requires the larger-memory machine class, so M3 Max Q4 numbers are `N/A`. ![M3 Max t/s](speed-bench/m3_max_ts.svg) ![PRO model M3 Ultra t/s](speed-bench/pro_model_m3_ultra_ts.svg) -## Running models larger than RAM +## 运行超出 RAM 容量的模型 -The normal Metal path tries to make the model resident in GPU-addressable -memory. This is the fastest path and should remain your default when the model -fits. When it does not fit, DwarfStar also has a Metal-only **SSD streaming** -capacity mode. In this mode the non-routed model weights stay resident, while -routed MoE experts are kept in an in-memory cache and loaded from the GGUF file -on cache misses. +常规 Metal 路径会尝试将模型常驻于 GPU 可寻址 +内存中。这是最快的路径，在模型 +能装入时应作为默认选择。若无法装入，DwarfStar 还提供仅 Metal 的 **SSD 流式传输（SSD streaming）** +容量模式。在此模式下，非路由（non-routed）模型权重保持常驻，而 +路由 MoE 专家保存在内存缓存中，缓存未命中时从 GGUF 文件 +加载。 -Streaming is not as fast as fitting the full model in RAM. It still needs memory -for non-routed weights, KV cache, graph scratch, activations, and the routed -expert cache. It is useful because routed experts dominate model size and modern -Mac SSDs are fast enough to make cache misses tolerable. Long prefills can still -be fast; generation is more sensitive to cache misses because every new token -routes through experts again. +流式传输不如将完整模型装入 RAM 快。它仍需要内存 +用于非路由权重、KV cache、图临时缓冲区、激活值以及路由 +专家缓存。之所以有用，是因为路由专家占模型体积的主体，而现代 +Mac SSD 足够快，使缓存未命中尚可接受。长预填充仍可 +很快；生成对缓存未命中更敏感，因为每个新 token +都会再次经过专家路由。 -Start with the automatic cache budget: +先从自动缓存预算开始： ```sh ./ds4 -m ./ds4flash.gguf --ssd-streaming ``` -If startup reports that the expert cache is too large, or if you want to reserve -more memory for context, set the routed expert cache explicitly: +若启动时报告专家缓存过大，或你想为上下文 +预留更多内存，请显式设置路由专家缓存： ```sh ./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB ``` -The `32GB` value is a memory budget for complete routed experts, not a generic -byte cache. DwarfStar converts it to the number of full experts that fit for the -current GGUF. Non-routed weights, KV cache, graph scratch, and activations need -additional memory. Only the automatic cache budget does the subtraction for you: -it takes 80% of the Metal recommended working set, subtracts non-routed weights, -then uses the rest for routed experts. Leave the hot expert preload enabled for -normal use; use `--ssd-streaming-cold` and `--ssd-streaming-preload-experts N` -only for measurements. +`32GB` 值是完整路由专家的内存预算，而非通用 +字节缓存。DwarfStar 会将其换算为当前 GGUF +可容纳的完整专家数量。非路由权重、KV cache、图临时缓冲区和激活值需要 +额外内存。只有自动缓存预算会替你完成扣减： +它取 Metal 推荐工作集的 80%，减去非路由权重， +余量用于路由专家。日常使用请保持热专家预加载开启；仅将 `--ssd-streaming-cold` 和 `--ssd-streaming-preload-experts N` +用于测量。 -### Practical SSD streaming examples +### SSD 流式传输实用示例 -On 64GB MacBooks, start with the 2-bit Flash GGUF and a moderate expert cache: +在 64GB MacBook 上，从 2-bit Flash GGUF 和适中的专家缓存开始： ```sh ./download_model.sh q2-imatrix @@ -230,8 +188,8 @@ On 64GB MacBooks, start with the 2-bit Flash GGUF and a moderate expert cache: --nothink ``` -On 128GB MacBooks, PRO q2 streaming is experimental but usable for inspection -and occasional work when you accept slow generation. Start with `--nothink`: +在 128GB MacBook 上，PRO q2 流式传输处于实验阶段，但若接受较慢的生成速度，仍可用于检视 +和偶尔使用。从 `--nothink` 开始： ```sh ./download_model.sh pro-q2-imatrix @@ -243,17 +201,16 @@ and occasional work when you accept slow generation. Start with `--nothink`: --nothink ``` -On an M5 Max with 128GB of RAM, a short PRO q2 streaming decode benchmark found -the automatic budget best: it selected about `59GB` of routed expert cache. -Manual `64GB` to `75GB` caches were close on that machine. Larger explicit -`NGB` requests are capped before inference so the expert buffers remain -lockable instead of falling into macOS paging. If the system is under extra -memory pressure and `mlock` still fails, ds4 refuses to install pageable -expert-cache entries and releases a locked-cache margin before continuing with -the measured lockable cache size. Prefer the automatic budget; if setting the -cache manually on this class of machine, start around `48GB` to `64GB`, then -increase only while the startup log reports a lockable cache. Once the machine -is stable, re-enable thinking with a conservative generation limit: +在一台 128GB RAM 的 M5 Max 上，简短的 PRO q2 流式解码基准测试表明 +自动预算最佳：它选定了约 `59GB` 的路由专家缓存。 +在该机器上，手动设置的 `64GB` 至 `75GB` 缓存与之接近。较大的显式 +`NGB` 请求会在推理前被上限封顶，以确保专家缓冲区保持 +可锁定状态，而非落入 macOS 分页。若系统承受额外 +内存压力且 `mlock` 仍失败，ds4 会拒绝安装可分页的 +专家缓存条目，并释放锁定缓存余量后再以 +实测的可锁定缓存大小继续运行。优先使用自动预算；若在此类机器上手动设置 +缓存，可从约 `48GB` 至 `64GB` 起步，然后 +仅在启动日志报告可锁定缓存时再逐步增大。机器稳定后，以保守的生成上限重新启用 thinking： ```sh ./ds4 \ @@ -264,42 +221,40 @@ is stable, re-enable thinking with a conservative generation limit: --tokens 1500 ``` -The important startup line is the cache report. Start conservative, then -increase the cache if the machine has headroom. +启动日志中关键的是缓存报告。从保守设置开始，若机器 +仍有余量再增大缓存。 -## Distributed Inference +## 分布式推理 -Distributed inference lets DwarfStar **run a model that is too large for one machine** by -splitting transformer layers across multiple machines. The main example is the -full 4-bit Flash quant across two 128 GB MacBooks: each process maps only its -own layer slice, activations are sent over TCP, and the coordinator keeps normal -CLI/API behavior. +分布式推理让 DwarfStar **在单台机器装不下的模型上运行**，方法是 +将 transformer 层拆分到多台机器上。主要示例是 +跨两台 128 GB MacBook 运行完整 4-bit Flash 量化：每个进程仅映射 +自己的层切片，激活值通过 TCP 传输，协调器保持常规 +CLI/API 行为。 -Distributed inference also allows to **speed up prefill** by -using multiple GPUs at the same time to process different micro-batches at -different layers, like in an assembly line. Only prefill can be accelerated this -way. Generation is purely autoregressive: each token must finish across the -route before the next token can start. The model work is the same as a single -process, plus coordination latency, so distributed generation is slower. +分布式推理还可**加速预填充**，方法是 +同时使用多台 GPU，在不同层处理不同微批次（micro-batch）， +如同流水线作业。仅预填充可借此加速。 +生成纯属自回归：每个 token 必须沿 +整条路由完成后，下一个 token 才能开始。模型计算量与单 +进程相同，还要加上协调延迟，因此分布式生成更慢。 -To build an initial mental model, here are the high level concepts: +要建立初步心智模型，以下是高层概念： -1. You put the GGUF on every machine, but each one loads just a subset. `--layers` controls which tensors are mapped, so a worker with `--layers 20:output` does not load the earlier layers. -2. Layer ranges are inclusive: `10:20` means layers 10, 11, ..., 20. `N:output` means layer `N` through the final layer plus the output head. -3. You assign one of the machines the role of `coordinator`, the others the roles of `workers`. Workers will connect to the coordinator and will tell they are there and which layers they are able to process. -4. Each worker keeps its slice of the KV cache. -5. Communication is worker-to-worker, there is no need to use the coordinator as relay, so if your coordinator is `A`, and you make a request, activations will flow in `A -> B -> C -> back to A`. +1. 你会在每台机器上都放置 GGUF，但每台机器只加载其中一部分。`--layers` 控制映射哪些张量，因此带有 `--layers 20:output` 的 worker 不会加载较早的层。 +2. 层范围是闭区间：`10:20` 表示第 10、11、…、20 层。`N:output` 表示从第 `N` 层到最后一层，外加输出头（output head）。 +3. 你将其中一台机器指定为 `coordinator` 角色，其余机器指定为 `workers` 角色。Worker 会连接到 coordinator，并告知自己已就绪以及能够处理哪些层。 +4. 每个 worker 保留其 KV cache 切片。 +5. 通信是 worker 到 worker 的，无需通过 coordinator 中继；因此若你的 coordinator 是 `A`，当你发起请求时，激活值会沿 `A -> B -> C -> back to A` 流动。 -### How it works and how to configure it +### 工作原理与配置方式 -The prefill path is pipelined (this is why it can go faster than in a single machine). -For large prompts the coordinator can run its -slice on chunk N+1 while the worker is running its slice on chunk N. The -distributed rows below were measured with two M5 Max 128 GB MacBooks connected -by Thunderbolt 5, using the Q4 Flash GGUF and the default 4096-token -distributed prefill chunk. The single-process column is a reference run with -the Q2 GGUF on a single machine, so it actually is a bit faster since -the routed MoEs are smaller. +预填充（prefill）路径采用流水线（pipeline）方式（这也是它比单机更快的原因）。 +对于长提示词，coordinator 可以在 worker 处理第 N 个分块（chunk）的切片时， +对其第 N+1 个分块运行自己的切片。下方分布式数据是在两台通过 Thunderbolt 5 连接的 +M5 Max 128 GB MacBook 上测得的，使用 Q4 Flash GGUF 和默认的 4096-token +分布式预填充分块。单机列是参考运行，使用 Q2 GGUF 在单机上运行，因此实际上会稍快一些， +因为路由的 MoE 更小。 | Prompt | Single-process reference | Two MacBooks | Speedup | | ---: | ---: | ---: | ---: | @@ -307,22 +262,17 @@ the routed MoEs are smaller. | 28684 tokens | 405.30 t/s | 674.16 t/s | 1.66x | | 63819 tokens | 353.62 t/s | 654.79 t/s | 1.85x | -Generation is different. **It is strictly autoregressive**: token N+1 cannot start -until token N has produced logits and sampling has selected the next token. That -means distributed generation cannot use the long prefill pipeline. It pays at -least one cross-machine activation hop per generated token, so generation is -slower than a single local process. On the same two-Mac Thunderbolt setup, a -12k-context control run with the 91 GB Flash quant went from 30.59 t/s -single-process to 24.67 t/s distributed, a 19.4% loss. Distributed inference is -therefore mainly for fitting larger models and speeding up long prefills, not -for making decode faster. +生成（generation）则不同。**它严格是自回归（autoregressive）的**：在 token N 产生 logits 且采样选出下一个 token 之前，token N+1 无法开始。 +这意味着分布式生成无法使用长预填充流水线。每个生成的 token 至少付出一次跨机器激活值跳转的代价， +因此生成速度会慢于单进程本地运行。在同一套双 Mac Thunderbolt 配置下，使用 91 GB Flash 量化的 12k 上下文对照运行， +单进程从 30.59 t/s 降至分布式 24.67 t/s，损失 19.4%。因此分布式推理主要用于 +容纳更大模型并加速长预填充，而非加快解码（decode）。 -### Full DeepSeek V4 PRO Q4 on two Mac Studios +### 在两台 Mac Studio 上运行完整 DeepSeek V4 PRO Q4 -The full-size PRO Q4 GGUF can be run across two 512 GB Mac Studio M3 Ultra -machines by giving the coordinator layers `0:30` and the worker -`31:output`. Use the split GGUF files so each side maps only the tensors it -needs: +完整尺寸的 PRO Q4 GGUF 可跨两台 512 GB Mac Studio M3 Ultra +机器运行：将 coordinator 配置为层 `0:30`，worker 配置为 +`31:output`。使用拆分后的 GGUF 文件，使每侧只映射其所需的张量： ```sh # Coordinator machine. @@ -332,35 +282,32 @@ needs: ./download_model.sh pro-q4-layers31-output ``` -The two files are: +两个文件为： ```text gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf ``` -This is a capacity use case: each process maps only its own half of the model, -while the worker owns the output head and returns logits. +这是一个容量用例：每个进程只映射模型自身的一半， +而 worker 拥有输出头并返回 logits。 -The current PRO Q4 Metal path uses queue-resident exact expert tables for the -large routed experts. This avoids the broad multi-GiB routed-tensor bindings -that made early distributed PRO Q4 attempts either run very slowly or hit Metal -memory accounting limits. In a short greedy smoke test over the direct -`192.168.0.182` / `192.168.0.183` link, the model generated coherent text and -measured 11.47 t/s generation after startup. Per-token telemetry was balanced: -local layers were around 39-43 ms, remote layers around 44-49 ms, for total -token times around 84-92 ms. Expect a slow startup while each side maps and -makes its half of the model resident. Long-context PRO Q4 prefill and decode -performance still needs separate benchmarking. +当前 PRO Q4 Metal 路径对大型路由专家使用队列驻留的精确专家表（exact expert tables）。 +这避免了宽泛的多 GiB 路由张量绑定，后者曾使早期分布式 PRO Q4 尝试要么运行极慢， +要么触及 Metal 内存核算限制。在直接 +`192.168.0.182` / `192.168.0.183` 链路上的简短贪心冒烟测试中，模型生成了连贯文本， +启动后测得 11.47 t/s 的生成速度。逐 token 遥测较为均衡： +本地层约 39–43 ms，远程层约 44–49 ms，总 token 时间约 84–92 ms。 +预计启动会较慢，因为每侧需要映射并使其那一半模型驻留。长上下文 PRO Q4 预填充与解码 +性能仍需单独基准测试。 -The measurements above use a Thunderbolt 5 cable. The implementation is plain -TCP and also works over slower links, including WiFi, but fast Ethernet or -Thunderbolt networking is strongly recommended. Slow links mostly hurt -generation latency and short prefills; large prefills can still benefit when -the layer split is balanced. In the normal performance path, the last worker -owns the output head and returns logits directly. +上述测量使用 Thunderbolt 5 线缆。实现采用普通 +TCP，在较慢链路上也能工作，包括 WiFi，但强烈建议使用高速以太网或 +Thunderbolt 网络。慢链路主要损害 +生成延迟和短预填充；当层划分均衡时，长预填充仍可能受益。在正常性能路径下，最后一个 worker +拥有输出头并直接返回 logits。 -Minimal two-host configuration: +最小双主机配置： ```sh # Machine A: coordinator, owns tokenization, sampling, the prompt, and layers 0..30. @@ -378,21 +325,18 @@ Minimal two-host configuration: --coordinator 169.254.43.68 1234 ``` -Normally the final worker should own the output head too, for example -`--layers 20:output`. This avoids returning a full final hidden-state batch -after prefill and lets the final worker produce the logits directly. On very -slow or metered links, `--layers 20:42` is also supported: the coordinator will -load the output head and compute logits locally, trading extra coordinator work -for smaller per-token replies. +通常最终 worker 也应拥有输出头，例如 +`--layers 20:output`。这可避免在预填充后返回完整的最终隐藏状态批次（hidden-state batch）， +并让最终 worker 直接产生 logits。在极慢或按量计费（metered）的链路上，也支持 `--layers 20:42`：coordinator 将 +加载输出头并在本地计算 logits，以额外的 coordinator 工作换取更小的逐 token 回复。 -### Network Link Comparison +### 网络链路对比 -The table below shows the same two M5 Max hosts, the same 91 GB Flash quant, -coordinator `--layers 0:19`, worker `--layers 20:output`, an 8192-token prompt -from `speed-bench/promessi_sposi.txt`, and 128 generated tokens. WiFi and -Internet numbers vary with local conditions, but the shape is the important -part: high latency hurts generation directly, while lower bandwidth also pulls -down long-prefill speed. +下表展示相同的两台 M5 Max 主机、相同的 91 GB Flash 量化、 +coordinator `--layers 0:19`、worker `--layers 20:output`、来自 `speed-bench/promessi_sposi.txt` 的 8192-token 提示词， +以及 128 个生成 token。WiFi 和 +Internet 数值会随本地条件变化，但形态才是关键： +高延迟会直接损害生成，而较低带宽也会拉低长预填充速度。 | Link | Addresses | Ping avg | Prefill | Generation | | --- | --- | ---: | ---: | ---: | @@ -400,18 +344,18 @@ down long-prefill speed. | WiFi | `192.168.1.57` -> `192.168.1.95` | 77.20 ms | 250.70 t/s | 10.70 t/s | | Internet / VPN | `10.77.0.4` -> `10.77.0.3` | 152.10 ms | 114.88 t/s | 3.63 t/s | -The Internet/VPN case is not meant to be a good interactive experience. It is -still useful for collective testing: multiple people can temporarily combine -machines to run a larger model that would not fit on any single host, accepting -slow decode in exchange for being able to inspect the model at all. +Internet/VPN 场景并非旨在提供良好的交互体验。它 +仍可用于集体测试：多人可临时组合 +机器以运行任何单主机都无法容纳的更大模型，接受 +缓慢的解码以换取至少能够检视该模型。 -Use the coordinator exactly like normal `./ds4`: interactive chat, `/read`, -and ordinary generation go through the same high-level session API. The same -distributed options are also wired into `ds4-agent`, `ds4-eval`, and -`ds4-bench`. For benchmarks, workers should already be running; `ds4-bench` -waits until a complete route is available. +请像正常使用 `./ds4` 一样使用 coordinator：交互式聊天、`/read`， +以及普通生成均通过相同的高层会话 API。相同的 +分布式选项也已接入 `ds4-agent`、`ds4-eval` 和 +`ds4-bench`。进行基准测试时，worker 应已在运行；`ds4-bench` +会等待直到完整路由可用。 -Useful tuning and diagnostics: +有用的调优与诊断： ```sh ./ds4-bench \ @@ -427,78 +371,34 @@ Useful tuning and diagnostics: --debug ``` -`--debug` on the coordinator prints route formation and per-hop telemetry: -layer range, token span, local evaluation time, downstream wait time, socket -send time, and input/output byte counts. This is the current profiling tool for -deciding whether a split is balanced. `--dist-prefill-window N` controls how -many prefill chunks may be in flight end-to-end; the default is conservative -and bounded. `--dist-prefill-chunk N` exists for experiments, but the default -4096-token chunk is the canonical setting and should be used unless you are -explicitly validating a different chunk size. +在 coordinator 上设置 `--debug` 会打印路由形成与逐跳遥测： +层范围、token 跨度、本地评估时间、下游等待时间、套接字 +发送时间，以及输入/输出字节数。这是当前用于 +判断划分是否均衡的分析工具。`--dist-prefill-window N` 控制 +端到端可同时在飞的预填充分块数量；默认较为保守 +且有界。`--dist-prefill-chunk N` 用于实验，但默认的 +4096-token 分块是规范设置，除非你明确在验证不同分块大小，否则应使用它。 -By default DwarfStar sends hidden-state activations as 32-bit floats. To reduce -traffic, pass `--dist-activation-bits 16` or `--dist-activation-bits 8` on the -coordinator. This changes only the transport format between machines, not the -model weights or KV cache. 16-bit transport halves activation traffic and is the -first option to try on Ethernet or WiFi. 8-bit transport is more aggressive and -should be treated as an approximate/experimental mode unless you have validated -the output for your use case. However experimentally reduction activation -size didn't provide a significant improvement, so this option may be removed -in the future. +默认情况下，DwarfStar 以 32 位浮点数发送隐藏状态激活值。为减少 +流量，可在 coordinator 上传入 `--dist-activation-bits 16` 或 `--dist-activation-bits 8`。 +这仅改变机器间的传输格式，不改变 +模型权重或 KV cache。16 位传输可将激活值流量减半，是在以太网或 WiFi 上应首先尝试的选项。8 位传输更为激进， +除非已针对你的用例验证输出，否则应视为近似/实验模式。然而实验表明缩小激活值 +尺寸并未带来显著改善，因此该选项未来可能会被移除。 -**If a worker disconnects, the coordinator removes that worker from the active -route**. The request already in flight can fail, and later calls report an -incomplete route until a compatible worker reconnects and sends a new -registration. For live sessions, the coordinator keeps the token history and can -rebuild worker KV state by replaying the prefix when the route is available -again. Workers also validate a rolling 64-bit token-prefix hash on every work -item, so a restarted worker at position 0 cannot silently accept work for -position N; it reports the mismatch and the coordinator replays the current -transcript. Ctrl+C in the CLI and agent is cooperative: DwarfStar waits for the -current distributed token or prefill chunk to drain before returning control, -which avoids coordinator-caused KV splits. Saved agent/server sessions use the -same KV file format as single-machine sessions: during save the coordinator -fetches worker-owned layer tensors and serializes one normal payload; during -load it splits that payload over the currently registered route. +**若有 worker 断开连接，coordinator 会将该 worker 从当前活跃路由中移除**。已在途中的请求可能会失败，后续调用会报告路由不完整，直到有兼容的 worker 重新连接并发送新的注册信息。对于实时会话，coordinator 会保留 token 历史，并在路由再次可用时通过重放前缀来重建 worker 的 KV 状态。Worker 还会在每个工作项上校验滚动的 64 位 token 前缀哈希（token-prefix hash），因此位于 position 0 的重启 worker 无法静默接受 position N 的工作；它会报告不匹配，coordinator 会重放当前 transcript。CLI 和 agent 中的 Ctrl+C 是协作式退出：DwarfStar 会等待当前分布式 token 或 prefill 块排空后再交还控制权，从而避免由 coordinator 导致的 KV 分裂。已保存的 agent/server 会话与单机会话使用相同的 KV 文件格式：保存时 coordinator 获取 worker 拥有的 layer 张量并序列化为一个常规 payload；加载时将该 payload 拆分分发到当前已注册的路由上。 -### Distributed protocol overview +### 分布式协议概览 -At the protocol level there are two kinds of connections. Workers keep a -control TCP connection open to the coordinator and send a `HELLO` with their -model ID, model family, quant profile, layer slice, context capacity, and data -port. The coordinator uses these registrations to build a route that covers all -layers. Work then moves over low-latency TCP data connections: the coordinator -computes the first slice, sends a `WORK` frame with session ID, token positions, -rolling token-prefix hashes before and after the span, route information, and -hidden-state payload, and each worker computes its slice. Middle workers can -forward directly to the next worker. The final worker returns logits to the -coordinator, or ACKs for non-final prefill chunks so the prefill pipeline can -stay full. `RESULT` frames echo the request ID and the post-span hash. A worker -status error is handled differently from a socket failure: KV/hash mismatch can -be recovered by replaying the token history on the same route, while transport -failure drops the route and waits for a replacement worker. For persistent KV, -the coordinator opens worker data connections and sends snapshot save/load -messages for each worker-owned layer range; the disk payload remains a single -agent/server cache file. The protocol has no -encryption or authentication, and is not release-stable yet; coordinator and -workers should be built from the same commit and used on trusted machines and -trusted networks. +在协议层面有两种连接。Worker 与 coordinator 保持一条控制用 TCP 连接，并发送携带 model ID、model family、quant profile、layer slice、context 容量和 data port 的 `HELLO`。Coordinator 用这些注册信息构建覆盖所有 layer 的路由。随后工作通过低延迟 TCP 数据连接传输：coordinator 计算第一个 slice，发送包含 session ID、token position、span 前后滚动 token 前缀哈希、路由信息和 hidden-state payload 的 `WORK` 帧，各 worker 计算自己的 slice。中间 worker 可直接转发给下一个 worker。最终 worker 将 logits 返回 coordinator，或对非最终的 prefill 块返回 ACK，以保持 prefill 流水线满载。`RESULT` 帧会回显 request ID 和 span 后的哈希。Worker 状态错误与 socket 故障的处理方式不同：KV/哈希不匹配可通过在同一路由上重放 token 历史恢复，而传输故障会丢弃该路由并等待替换 worker。对于持久化 KV，coordinator 打开 worker 数据连接，为每个 worker 拥有的 layer 范围发送快照保存/加载消息；磁盘 payload 仍是一个 agent/server 缓存文件。该协议尚无加密或认证，且尚未达到发布级稳定性；coordinator 与 worker 应从同一 commit 构建，并仅在可信机器与可信网络上使用。 -## Reducing heat, power usage and fan noise +## 降低发热、功耗与风扇噪声 -Long local inference runs can keep the GPU busy for extended periods. If you -care more about heat, fan noise, battery life on MacBooks, or reducing thermal -stress on the hardware than about maximum throughput, use `--power N`. +长时间本地推理会让 GPU 长时间满载。若你更关心发热、风扇噪声、MacBook 续航，或降低硬件热应力，而非追求最大吞吐量，请使用 `--power N`。 -`--power 100` is the default and means full speed. Lower values ask DwarfStar to target -that percentage of GPU usage: `--power 70` targets about 70%, `--power 50` -targets about half usage, and so forth. DwarfStar does this by measuring GPU work time -and inserting small sleeps between work units: during prefill it sleeps between -layers, and during generation it sleeps between decoded tokens. This reduces -sustained load without changing model output. +`--power 100` 为默认值，表示全速运行。更低的数值会让 DwarfStar 将 GPU 使用率目标设为该百分比：`--power 70` 约 70%，`--power 50` 约一半，以此类推。DwarfStar 通过测量 GPU 工作时间并在工作单元之间插入短暂休眠来实现：prefill 时在 layer 之间休眠，generation 时在解码 token 之间休眠。这样可降低持续负载，且不会改变模型输出。 -The option is available on the CLI, server, agent, eval, and benchmark tools, -for example: +该选项适用于 CLI、server、agent、eval 和 benchmark 工具，例如： ```sh ./ds4 --power 50 @@ -506,46 +406,26 @@ for example: ./ds4-server --power 40 --ctx 100000 ``` -## Native agent +## 原生 agent -DwarfStar features a native coding agent that works in a different way -than most other systems: the inference is controlled from within the agent -itself, without socket/API boundaries, so the session is represented -by the on-disk KV cache itself. Moreover the tools and the system prompt -are all designed vertically for DeepSeek v4 Flash and PRO. This provides a -few advantages: +DwarfStar 提供原生编程 agent，工作方式与多数系统不同：推理由 agent 内部直接控制，无 socket/API 边界，会话由磁盘上的 KV cache 本身表示。此外，tools 与 system prompt 均针对 DeepSeek v4 Flash 和 PRO 垂直设计。这带来若干优势： -* Low latency experience, bounded mainly by the prefill speed limits. Displaying of generated text, tool calling, start of a new session are always instantaneous. -* Live progress bar during prefill time. -* No DSML tool calling conversion, the tools are handled natively in the LLM format. -* KV cache mismatch are impossible by construction, the current state is always the truth. -* Everything is tuned for this model. -* Ability to switch saved sessions with `/list` and `/switch`; full KV sessions resume without a prefill stage. +* 低延迟体验，主要受 prefill 速度限制。生成文本显示、tool 调用、新会话启动均为即时。 +* prefill 期间有实时进度条。 +* 无需 DSML tool 调用转换，tools 以 LLM 格式原生处理。 +* 按设计不可能出现 KV cache 不匹配，当前状态即为权威真相。 +* 一切针对该模型调优。 +* 可用 `/list` 和 `/switch` 切换已保存会话；完整 KV 会话恢复时无需 prefill 阶段。 -Agent sessions are stored in `~/.ds4/kvcache`. Use `/save` to persist the -current session, `/list` to show saved sessions sorted by recent update time, -and `/switch ` to resume one of them. The session ID is stable across -future saves and is derived from the first user prompt and creation time. -`/del ` removes a saved session. `/strip ` keeps the rendered -conversation text and title but removes the heavy KV payload; switching to a -stripped session rebuilds the KV cache by prefilling the saved text. +Agent 会话存储在 `~/.ds4/kvcache`。使用 `/save` 持久化当前会话，`/list` 按最近更新时间列出已保存会话，`/switch ` 恢复其中一个。会话 ID 在后续保存中保持稳定，由首次用户 prompt 与创建时间派生。`/del ` 删除已保存会话。`/strip ` 保留已渲染的对话文本与标题，但移除沉重的 KV payload；切换到精简会话时，通过对已保存文本进行 prefill 重建 KV cache。 -Use `--chdir /path/to/ds4` when launching `ds4-agent` from another directory, -so relative runtime files such as `metal/*.metal` resolve from the project tree. +从其他目录启动 `ds4-agent` 时，请使用 `--chdir /path/to/ds4`，以便 `metal/*.metal` 等相对运行时文件从项目目录解析。 -However while the system already works, there is a lot of work to do -in order to make it ready for prime time. When finally the agent will reach -the wanted shape, we will *likely* split the server and the client creating a stateful -session-based protocol that can recreate all that in a client-server way. +不过系统虽已可用，要达到生产就绪仍需大量工作。当 agent 最终达到预期形态时，我们*可能*会拆分 server 与 client，建立有状态、基于会话的协议，以 client-server 方式复现上述能力。 -## Benchmarking +## 基准测试 -`ds4-bench` measures instantaneous prefill and generation throughput at context -frontiers instead of reporting one whole-run average. It loads the model once, -walks a fixed token sequence to frontiers such as 2048, 4096, 6144, and uses -incremental prefill so each row measures only the newly-added token interval. -After each frontier it saves the live KV state to memory, generates a fixed -greedy non-EOS probe, restores the memory snapshot, and continues prefill. +`ds4-bench` 在 context 边界处测量瞬时 prefill 与 generation 吞吐量，而非报告整次运行的平均值。它一次性加载模型，沿固定 token 序列推进至 2048、4096、6144 等边界，并使用增量 prefill，使每一行仅测量新增 token 区间。每个边界后，它将实时 KV 状态保存到内存，生成固定贪心非 EOS 探测，恢复内存快照，然后继续 prefill。 ```sh ./ds4-bench \ @@ -557,55 +437,25 @@ greedy non-EOS probe, restores the memory snapshot, and continues prefill. --gen-tokens 128 ``` -The example file is a cleaned public-domain Project Gutenberg text of -Alessandro Manzoni's *I Promessi Sposi* (ebook #45334), with the Gutenberg -header and footer removed: . +示例文件是经清理的公有领域 Project Gutenberg 文本，内容为 Alessandro Manzoni 的 *I Promessi Sposi*（电子书 #45334），已移除 Gutenberg 页眉页脚：。 -Use `--step-incr N` for different linear spacing, or `--step-mul F` for -exponential sweeps. Output is CSV with one row per frontier: latest prefill -interval tokens/sec, generation tokens/sec at that frontier, and -`kvcache_bytes`. +使用 `--step-incr N` 设置不同线性间隔，或使用 `--step-mul F` 进行指数扫描。输出为 CSV，每个边界一行：最近 prefill 区间的 tokens/sec、该边界处的 generation tokens/sec，以及 `kvcache_bytes`。 -Sessions prefill long prompts in 4096-token chunks by default. Set -`DS4_METAL_PREFILL_CHUNK=N` to compare another chunk size, for example `2048` -to match the strict official-vector checkpoint path, or -`DS4_METAL_PREFILL_CHUNK=0` to prefill a prompt as one whole batch when memory -allows. Changing the chunk changes the KV checkpoint/logit path, so compare it -as an explicit run configuration. -Chunked Metal prefill reuses the same range-capable layer-major graph for each -chunk, preserving absolute compressor/indexer boundaries while avoiding the old -per-layer chunk dispatch path. +会话默认以 4096 token 块对长 prompt 进行 prefill。设置 `DS4_METAL_PREFILL_CHUNK=N` 以对比其他块大小，例如 `2048` 以匹配严格 official-vector checkpoint 路径，或 `DS4_METAL_PREFILL_CHUNK=0` 在内存允许时将 prompt 作为整批 prefill。更改块大小会改变 KV checkpoint/logit 路径，因此应作为显式运行配置进行对比。分块 Metal prefill 为每个块复用同一支持 range 的 layer-major 图，保留绝对 compressor/indexer 边界，同时避免旧的逐层分块调度路径。 -## Capability Evaluation +## 能力评估 -`ds4-eval` is a small real-model integration benchmark. It is not a leaderboard -runner and should not be reported as an official GPQA, SuperGPQA, AIME, or -security benchmark score: the questions are an embedded 92-item subset chosen -to make local regression testing useful and visually inspectable. The program -loads the real GGUF, renders DeepSeek chat prompts, streams sampled tokens in a split-screen TUI, grades -the final answer, and prints a per-question report with prompt tokens, -generated tokens, pass/fail state, the model answer, and the correct answer. +`ds4-eval` 是小型真实模型集成基准测试。它不是排行榜运行器，不应作为官方 GPQA、SuperGPQA、AIME 或安全基准分数上报：题目为内嵌 92 题子集，旨在使本地回归测试实用且便于目视检查。程序加载真实 GGUF，渲染 DeepSeek chat prompt，在分屏 TUI 中流式输出采样 token，评判最终答案，并打印逐题报告，含 prompt token 数、生成 token 数、通过/失败状态、模型答案与正确答案。 ```sh ./ds4-eval -m ds4flash.gguf --trace /tmp/ds4-eval.txt ``` -The default run uses `--tokens 16000`, thinking mode enabled, and a soft/hard -`` budget cutoff so the model has room to produce a visible answer. -`ds4-eval` sizes the context internally from the largest selected prompt plus -the generation budget, and refuses runs that would need more than 1M context -tokens. Press `p` to pause, `q` to exit and print the report, Up/Down to -inspect or select another question, and Enter to run the selected question next. -`--plain` disables the TUI. +默认运行使用 `--tokens 16000`，启用 thinking mode，并设置 soft/hard `` 预算截止，以便模型有足够空间生成可见答案。`ds4-eval` 根据所选最大 prompt 加 generation 预算在内部确定 context 大小，并拒绝需要超过 1M context token 的运行。按 `p` 暂停，`q` 退出并打印报告，Up/Down 查看或选择其他题目，Enter 运行下一道所选题目。`--plain` 可禁用 TUI。 -Use `--regrade-trace /path/to/trace.txt` to replay the current answer -extractor and scorer against a prior `--trace` file without loading the model -or regenerating tokens. This is useful when auditing evaluator changes: it -shows which cases changed, the old picked answer, the new picked answer, and a -pass/fail summary. +使用 `--regrade-trace /path/to/trace.txt` 可针对先前的 `--trace` 文件重放当前的答案提取器与评分器，而无需加载模型或重新生成 token。这在审计评估器变更时很有用：它会显示哪些用例发生了变化、原先选中的答案、新选中的答案，以及通过/失败摘要。 -For inference changes that can affect generation drift, keep this deterministic -q1..q4 token-count gate in the test plan: +对于可能影响生成漂移（generation drift）的推理变更，请在测试计划中保留这一确定性的 q1..q4 token 数量门控： ```sh ./ds4-eval \ @@ -617,7 +467,7 @@ q1..q4 token-count gate in the test plan: --seed 1 ``` -The generated-token counts must stay aligned with the baseline: +生成的 token 数量必须与基线保持一致： | Question | Expected state | Expected generated tokens | Expected given/correct | |---:|---|---:|---| @@ -626,95 +476,51 @@ The generated-token counts must stay aligned with the baseline: | 3 | `PASSED` | 666 | `70` / `70` | | 4 | `FAILED` | 2048 | `A` / `C` | -The first 75 embedded questions are interleaved as 25 GPQA Diamond, 25 audited -SuperGPQA, and 25 AIME 2025 problems. The final 17 are an audited COMPSEC -subset of reduced single-function C/C++ vulnerability-localization questions. -The model is asked for the single best source line, or the smallest exact line -set only when the bug cannot be localized to one line; the scorer accepts small -audited ranges only when adjacent lines are equivalent locations for the same -bug. The order is -intentionally progressive: early questions are useful smoke tests, while later -questions are hard enough that a strong reasoning model should still miss some -of them. The SuperGPQA slice is curated rather than blind: upstream rows with -wrong keys, missing figures, or underspecified prompts are replaced with cleaner -rows. +前 75 道内嵌题目按 25 道 GPQA Diamond、25 道经审计的 SuperGPQA 和 25 道 AIME 2025 题目交错排列。最后 17 道是经审计的 COMPSEC 子集，为精简后的单函数 C/C++ 漏洞定位题。模型需给出单一最佳源码行；仅当缺陷无法定位到一行时，才返回最小的精确行集合；评分器仅在相邻行对同一缺陷属于等价位置时，才接受小的经审计范围。题目顺序刻意递进：早期题目适合作为冒烟测试，后期题目足够难，强推理模型仍应会漏掉其中一部分。SuperGPQA 切片经过人工筛选而非盲抽：上游数据中键值错误、缺少图示或提示不明确的条目，会替换为更干净的行。 -The set should be treated as a hard capability regression suite rather than -a pass/fail unit test. +该集合应视为硬性能力回归套件，而非通过/失败的单元测试。 -- **GPQA Diamond** contributes graduate-level science questions with - multiple-choice answers. DeepSeek's model card reports strong results - on full GPQA Diamond in thinking mode, but individual items still require - careful physics, chemistry, or biology reasoning and are easy to lose with a - small prompt/rendering or sampling regression. -- **SuperGPQA** contributes broad specialist knowledge and domain-transfer - questions. The model-card SuperGPQA number is much lower than GPQA Diamond, - so these items are expected to be uneven: some look mundane, others require - niche professional knowledge or exact interpretation of a translated-style - exam question. -- **AIME 2025** contributes exact-answer contest math. These are often the most - unforgiving items in the set: no multiple-choice prior, no partial credit, and - a single arithmetic or algebraic slip changes the grade. -- **COMPSEC** contributes single-function C/C++ security reasoning items - reduced from public CVE writeups. These are not exploit prompts: the task is - to identify the best source line where the defensive code flaw is introduced, - or return `0` for a safe function. +- **GPQA Diamond** 提供研究生水平的科学选择题。DeepSeek 的模型卡（model card）报告称在 thinking 模式下于完整 GPQA Diamond 上表现强劲，但个别题目仍需要仔细的物理、化学或生物学推理，且容易因提示/渲染或采样方面的微小回归而失分。 +- **SuperGPQA** 提供广泛的专业知识与跨领域迁移题。模型卡上的 SuperGPQA 分数远低于 GPQA Diamond，因此这些题目预期表现不均：有些看似平常，另一些则需要小众专业知识，或对翻译风格考试题的精确解读。 +- **AIME 2025** 提供要求精确答案的竞赛数学题。这些往往是本套件中最不容错的题目：没有多选先验、没有部分得分，一次算术或代数失误就会改变成绩。 +- **COMPSEC** 提供从公开 CVE 分析报告中精简而来的单函数 C/C++ 安全推理题。这些不是漏洞利用提示：任务是识别引入防御性代码缺陷的最佳源码行，或对安全函数返回 `0`。 -In practice this means `ds4-eval` should not be expected to produce a perfect -92/92 run. It is meant to answer a more useful engineering question: after a -kernel, quantization, prompt-rendering, KV-cache, or tool-streaming change, does -DeepSeek V4 Flash still solve a representative mix of hard science, broad -knowledge, exact math, and security-code problems while using the same inference -path users run? +实践中这意味着不应期望 `ds4-eval` 能跑出完美的 92/92。它旨在回答一个更有用的工程问题：在内核、量化、提示渲染、KV-cache 或工具流式传输变更之后，DeepSeek V4 Flash 是否仍能在与用户相同的推理路径下，解决涵盖硬科学、广博知识、精确数学与安全代码问题的代表性混合集？ ## CLI -One-shot prompt: +单次提示： ```sh ./ds4 -p "Explain Redis streams in one paragraph." ``` -No `-p` starts the interactive prompt: +不带 `-p` 可启动交互式提示： ```sh ./ds4 ds4> ``` -The interactive CLI is a real multi-turn chat. It keeps the rendered chat -transcript and the live graph KV checkpoint, so each turn extends the previous -conversation. Useful commands are `/help`, `/think`, `/think-max`, `/nothink`, -`/ctx N`, `/read FILE`, and `/quit`. Ctrl+C interrupts the current generation -and returns to `ds4>`. +交互式 CLI 是真正的多轮对话。它会保留已渲染的聊天记录与实时的 graph KV 检查点，因此每一轮都会延续上一轮对话。实用命令包括 `/help`、`/think`、`/think-max`、`/nothink`、`/ctx N`、`/read FILE` 和 `/quit`。Ctrl+C 会中断当前生成并返回到 `ds4>`。 -The CLI defaults to thinking mode. Use `/nothink` or `--nothink` for direct -answers. `--mtp MTP.gguf --mtp-draft 2` enables the optional MTP speculative -path; it is useful only for greedy decoding, currently uses a confidence gate -(`--mtp-margin`) to avoid slow partial accepts, and should be treated as an -experimental slight-speedup path. +CLI 默认使用 thinking 模式。使用 `/nothink` 或 `--nothink` 可直接作答。`--mtp MTP.gguf --mtp-draft 2` 启用可选的 MTP 推测路径；它仅对贪心解码有用，目前使用置信度门控（`--mtp-margin`）以避免缓慢的局部接受，应视为实验性的轻微加速路径。 ## Server -Start a local OpenAI/Anthropic-compatible server: +启动本地 OpenAI/Anthropic 兼容服务器： ```sh ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 ``` -Use `--chdir /path/to/ds4` when launching `ds4-server` from another directory, -so relative runtime files such as `metal/*.metal` resolve from the project tree. +从其他目录启动 `ds4-server` 时，请使用 `--chdir /path/to/ds4`，以便 `metal/*.metal` 等相对运行时文件能从项目目录正确解析。 -The server keeps one mutable backend/KV checkpoint in memory, -so stateless clients that resend a longer version of the same prompt can reuse -the shared prefix instead of pre-filling from token zero. +服务器在内存中保留一个可变的 backend/KV 检查点，因此无状态客户端若重新发送同一提示的更长版本，可复用共享前缀，而无需从 token 零开始预填充。 -Request parsing and sockets run in client threads, but inference itself is -serialized through one graph worker. The current server does not batch multiple -independent requests together; concurrent requests wait their turn on the single -live graph/session. +请求解析与套接字在客户端线程中运行，但推理本身通过单个 graph worker 串行化。当前服务器不会将多个独立请求批量处理；并发请求需在单一活跃 graph/会话上依次等待。 -Supported endpoints: +支持的端点： - `GET /v1/models` - `GET /v1/models/deepseek-v4-flash` @@ -724,83 +530,31 @@ Supported endpoints: - `POST /v1/completions` - `POST /v1/messages` -The Flash and PRO model endpoints are compatibility aliases. They both report -the model currently loaded from the GGUF passed with `-m`; the endpoint name does -not select a different model. +Flash 与 PRO 模型端点是兼容性别名。二者均报告通过 `-m` 传入的 GGUF 当前加载的模型；端点名称不会选择不同模型。 -`/v1/chat/completions` accepts the usual OpenAI-style `messages`, -`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, `top_k`, `min_p`, -`seed`, `stream`, `stream_options.include_usage`, `tools`, and `tool_choice`. -Tool schemas are rendered into DeepSeek's DSML tool format, and generated DSML -tool calls are mapped back to OpenAI tool calls. +`/v1/chat/completions` 接受常见的 OpenAI 风格 `messages`、`max_tokens`/`max_completion_tokens`、`temperature`、`top_p`、`top_k`、`min_p`、`seed`、`stream`、`stream_options.include_usage`、`tools` 和 `tool_choice`。工具 schema 会渲染为 DeepSeek 的 DSML 工具格式，生成的 DSML 工具调用会映射回 OpenAI 工具调用。 -`/v1/responses` accepts OpenAI Responses-style `input`, `instructions`, -`tools`, `tool_choice`, `max_output_tokens`, `temperature`, `top_p`, `stream`, -and `reasoning`. It is the preferred endpoint for Codex CLI. The server keeps -Responses continuations bound to live state when possible, and can fall back to -the same DSML rendering and KV prefix reuse used by chat completions. +`/v1/responses` 接受 OpenAI Responses 风格的 `input`、`instructions`、`tools`、`tool_choice`、`max_output_tokens`、`temperature`、`top_p`、`stream` 和 `reasoning`。它是 Codex CLI 的首选端点。服务器会尽可能将 Responses 续写绑定到实时状态，并可回退到与 chat completions 相同的 DSML 渲染与 KV 前缀复用。 -`/v1/messages` is the Anthropic-compatible endpoint used by Claude Code style -clients. It accepts `system`, `messages`, `tools`, `tool_choice`, `max_tokens`, -`temperature`, `top_p`, `top_k`, `stream`, `stop_sequences`, and thinking -controls. Tool uses are returned as Anthropic `tool_use` blocks. +`/v1/messages` 是 Claude Code 风格客户端使用的 Anthropic 兼容端点。它接受 `system`、`messages`、`tools`、`tool_choice`、`max_tokens`、`temperature`、`top_p`、`top_k`、`stream`、`stop_sequences` 以及 thinking 控制项。工具使用以 Anthropic `tool_use` 块形式返回。 -Default sampled API generation uses `temperature=1`, `top_p=1`, and -`min_p=0.05`, so the default filter is relative probability rather than -nucleus mass. In thinking mode DwarfStar uses those fixed sampling defaults and -ignores client sampling knobs, matching DeepSeek's fixed-thinking API behavior. +默认采样 API 生成使用 `temperature=1`、`top_p=1` 和 `min_p=0.05`，因此默认过滤器基于相对概率而非 nucleus 质量。在 thinking 模式下，DwarfStar 使用这些固定采样默认值并忽略客户端采样旋钮，与 DeepSeek 固定 thinking API 行为一致。 -The chat, Responses, and Anthropic endpoints support SSE streaming. In thinking -mode, reasoning is streamed in the native API shape instead of being mixed into -final text. OpenAI chat streaming -also streams tool calls as soon as the DSML invocation is recognized: the tool -header is sent first, then parameter bytes are forwarded as -`tool_calls[].function.arguments` deltas while generation continues. The -Anthropic endpoint streams thinking and text live, then emits structured -`tool_use` blocks when the generated tool block is complete. -The Responses endpoint streams the Responses event lifecycle expected by Codex, -including `response.output_text.delta`, function-call argument events, and -terminal `response.completed` / `response.incomplete` / `response.failed` -events. +chat、Responses 与 Anthropic 端点均支持 SSE 流式传输。在 thinking 模式下，推理以原生 API 形态流式输出，而非混入最终文本。OpenAI chat 流式传输还会在识别到 DSML 调用后立即流式输出工具调用：先发送工具头，随后在生成继续时将参数字节作为 `tool_calls[].function.arguments` 增量转发。Anthropic 端点会实时流式输出 thinking 与文本，并在生成的工具块完成后发出结构化的 `tool_use` 块。Responses 端点会流式输出 Codex 所期望的 Responses 事件生命周期，包括 `response.output_text.delta`、函数调用参数事件，以及终止性的 `response.completed` / `response.incomplete` / `response.failed` 事件。 -For browser JavaScript clients served from another origin, start the server with -`--cors` to emit `Access-Control-Allow-*` headers. This only changes HTTP -headers; it does not expose the server on the LAN. Use `--host 0.0.0.0` -explicitly when remote machines should be able to connect. +对于从其他源站提供服务的浏览器 JavaScript 客户端，请使用 `--cors` 启动服务器以发出 `Access-Control-Allow-*` 头。这仅更改 HTTP 头；并不会将服务器暴露到局域网。若需远程机器能够连接，请显式使用 `--host 0.0.0.0`。 ### Tool call handling and canonicalization -DeepSeek V4 emits tool calls as [DSML text](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md). Agent clients do not send that -same text back on the next request: they send normalized OpenAI/Anthropic JSON -tool-call objects. **If the server re-rendered those objects slightly -differently, the rendered byte prefix would no longer match the live KV -checkpoint** and the next turn would have to be rebuilt. +DeepSeek V4 以 [DSML text](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md).) 形式发出工具调用。Agent 客户端在下次请求中不会发回相同文本：它们发送规范化的 OpenAI/Anthropic JSON 工具调用对象。**若服务器将这些对象重新渲染时略有不同，渲染后的字节前缀将不再与实时 KV 检查点匹配**，下一轮对话就必须重建。 -The first line of defense is exact replay. Every tool call gets an unguessable -API tool ID, and the server remembers `tool id -> exact sampled DSML block` in -a bounded in-memory map backed by radix trees. When the client later sends that -tool ID back, the prompt renderer uses the exact DSML bytes the model sampled, -not a freshly formatted approximation. This map can also be saved inside KV -cache files, so exact replay survives server restarts for cached histories. +第一道防线是精确重放（exact replay）。每次工具调用都会获得一个无法猜测的 API 工具 ID，服务器会在由基数树（radix trees）支撑的有限内存映射中记住 `tool id -> exact sampled DSML block`。当客户端稍后将该工具 ID 发回时，提示词渲染器会使用模型采样到的精确 DSML 字节，而不是新格式化的近似值。该映射也可保存在 KV 缓存文件中，因此对于已缓存的历史记录，精确重放可在服务器重启后依然有效。 -**Canonicalization is only the backup path**. If the exact DSML block is missing, -or exact replay is disabled with `--disable-exact-dsml-tool-replay`, the server -renders a deterministic DSML form from the JSON tool object. After a tool-call -turn, it compares the live sampled token stream with the prompt that the next -client request will render. If needed, it rewrites the live checkpoint, or -falls back to an older disk KV snapshot and replays only the suffix. This keeps -the model continuation aligned with the stateless API transcript. +**规范化（Canonicalization）仅是备用路径**。如果精确的 DSML 块缺失，或通过 `--disable-exact-dsml-tool-replay` 禁用了精确重放，服务器会从 JSON 工具对象渲染确定性的 DSML 形式。在工具调用轮次结束后，它会将实时采样的 token 流与下一次客户端请求将渲染的提示词进行比较。如有必要，它会重写实时检查点，或回退到较早的磁盘 KV 快照并仅重放后缀部分。这使模型续写与无状态 API 对话记录保持一致。 -During generation, the server also treats DSML syntax differently from payload. -When the model is emitting stable protocol structure such as DSML tags, -parameter headers, JSON punctuation, or closing markers, sampling is forced to -`temperature=0` so the tool call stays parseable. This greedy mode does **not** -apply to argument payloads: `string=true` parameter bodies and JSON string -values, including file contents and edit text, use the request's normal sampling -settings. That separation is important: deterministic decoding is helpful for -syntax, but can create repeated text when applied to long code or file bodies. +在生成过程中，服务器对 DSML 语法与载荷（payload）采用不同处理。当模型输出稳定的协议结构（如 DSML 标签、参数头、JSON 标点或闭合标记）时，采样被强制设为 `temperature=0`，以确保工具调用可被解析。这种贪婪模式**不**适用于参数载荷：`string=true` 参数体与 JSON 字符串值（包括文件内容和编辑文本）使用请求的正常采样设置。这种分离很重要：确定性解码对语法有帮助，但应用于长代码或文件体时可能产生重复文本。 -Minimal OpenAI example: +最小 OpenAI 示例： ```sh curl http://127.0.0.1:8000/v1/chat/completions \ @@ -812,31 +566,19 @@ curl http://127.0.0.1:8000/v1/chat/completions \ }' ``` -### Agent Client Usage +### 代理客户端用法 -`ds4-server` can be used by local coding agents that speak OpenAI-compatible -chat completions. Start the server first, and set the client context limit no -higher than the `--ctx` value you started the server with: +`ds4-server` 可供支持 OpenAI 兼容 chat completions 的本地编码代理使用。先启动服务器，并将客户端上下文限制设置为不高于你启动服务器时使用的 `--ctx` 值： ```sh ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 ``` -You can use larger context and larger cache if you wish. Full context of -1M tokens is going to use more or less 26GB of memory (compressed indexer -alone will be like 22GB), so configure a context which makes sense in -your system. With 128GB of RAM you would run the 2-bit quants, which are -already 81GB, 26GB are going to be likely too much, so a context window -of 100~300k tokens is wiser. However users reported being able to run 2bit -quants with 250k ctx window in a Macs with just 96GB of system memory: make sure -to kill processes that use too much memory, if you plan doing so ;) +如需可使用更大的上下文和更大的缓存。完整的 1M token 上下文大约需要 26GB 内存（仅压缩索引器就约 22GB），因此请根据你的系统配置合理的上下文大小。若拥有 128GB 内存，你可能会运行 2-bit 量化模型（本身已占 81GB），额外 26GB 可能过多，因此 100~300k token 的上下文窗口更明智。不过有用户报告称，在仅 96GB 系统内存的 Mac 上也能以 250k ctx 窗口运行 2bit 量化模型：若你打算这样做，务必结束占用过多内存的进程 ;) -The `384000` output limit below avoids token caps since the model is able -to generate very long replies otherwise (up to 384k tokens). The server -still stops when the configured context window is full. +下面的 `384000` 输出限制可避免 token 上限，否则模型可能生成非常长的回复（最多 384k token）。当配置的上下文窗口已满时，服务器仍会停止。 -For **opencode**, add a provider and agent entry to -`~/.config/opencode/opencode.json`: +对于 **opencode**，在 `~/.config/opencode/opencode.json` 中添加 provider 和 agent 条目： ```json { @@ -870,7 +612,7 @@ For **opencode**, add a provider and agent entry to } ``` -For **Pi**, add a provider to `~/.pi/agent/models.json`: +对于 **Pi**，在 `~/.pi/agent/models.json` 中添加 provider： ```json { @@ -919,7 +661,7 @@ For **Pi**, add a provider to `~/.pi/agent/models.json`: } ``` -Optionally make it the default Pi model in `~/.pi/agent/settings.json`: +可选：在 `~/.pi/agent/settings.json` 中将其设为默认 Pi 模型： ```json { @@ -928,7 +670,7 @@ Optionally make it the default Pi model in `~/.pi/agent/settings.json`: } ``` -For **Codex CLI**, use the Responses wire API: +对于 **Codex CLI**，使用 Responses wire API： ```toml [model_providers.ds4] @@ -938,14 +680,13 @@ wire_api = "responses" stream_idle_timeout_ms = 1000000 ``` -Then run: +然后运行： ```sh codex --model deepseek-v4-flash -c model_provider=ds4 ``` -For **Claude Code**, use the Anthropic-compatible endpoint. A wrapper like this -matches the local `~/bin/claude-ds4` setup: +对于 **Claude Code**，使用 Anthropic 兼容端点。如下包装器与本地 `~/bin/claude-ds4` 配置匹配： ```sh #!/bin/sh @@ -971,60 +712,33 @@ export CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000 exec "$HOME/.local/bin/claude" "$@" ``` -Claude Code may send a large initial prompt, often around 25k tokens, before it -starts doing useful work. Keep `--kv-disk-dir` enabled: after the first expensive -prefill, the disk KV cache lets later continuations or restarted sessions reuse -the saved prefix instead of processing the whole prompt again. +Claude Code 在开始执行有用工作前可能会发送很大的初始提示词，通常约 25k token。请保持 `--kv-disk-dir` 启用：在首次昂贵的预填充（prefill）之后，磁盘 KV 缓存可让后续续写或重启的会话复用已保存的前缀，而无需再次处理整个提示词。 -## Thinking Modes +## 思考模式 -DeepSeek V4 Flash has distinct non-thinking, thinking, and Think Max modes. -The server defaults to thinking mode. `reasoning_effort=max` requests Think -Max, but it is only applied when the context size is large enough for the model -card recommendation; smaller contexts fall back to normal thinking. OpenAI -`reasoning_effort=xhigh` still maps to normal thinking, not Think Max. +DeepSeek V4 Flash 具有不同的非思考（non-thinking）、思考（thinking）和 Think Max 模式。服务器默认使用思考模式。`reasoning_effort=max` 请求 Think Max，但仅在上下文大小达到模型卡推荐要求时才会应用；较小的上下文会回退到普通思考模式。OpenAI 的 `reasoning_effort=xhigh` 仍映射到普通思考模式，而非 Think Max。 -For direct replies, use `thinking: {"type":"disabled"}`, `think:false`, or a -non-thinking model alias such as `deepseek-chat`. +如需直接回复，请使用 `thinking: {"type":"disabled"}`、`think:false`，或非思考模型别名（如 `deepseek-chat`）。 -## Disk KV Cache +## 磁盘 KV 缓存 -Chat/completion APIs are stateless: agent clients usually resend the whole -conversation every request. `ds4-server` first tries the cheap exact token-prefix -check, then falls back to comparing rendered prompt bytes with decoded -checkpoint bytes. The live in-memory checkpoint covers the current session; the -disk KV cache makes useful prefixes survive session switches and server -restarts. +Chat/completion API 是无状态的：代理客户端通常每次请求都会重发整个对话。`ds4-server` 首先尝试廉价的精确 token 前缀检查，然后回退到比较已渲染的提示词字节与解码后的检查点字节。实时内存检查点覆盖当前会话；磁盘 KV 缓存使有用的前缀在会话切换和服务器重启后得以保留。 -For RAM reasons there is currently only one live KV cache in memory. When a new -unrelated session replaces it, the old checkpoint can only be resumed without -re-processing if it was written to the disk KV cache. In other words, memory -cache handles the active session; disk cache is the resume mechanism for -different sessions. +出于内存原因，当前内存中仅有一个实时 KV 缓存。当新的无关会话将其替换时，旧检查点只有在已写入磁盘 KV 缓存时才能在不重新处理的情况下恢复。换言之，内存缓存处理活动会话；磁盘缓存是不同会话的恢复机制。 -Enable it with: +通过以下方式启用： ```sh ./ds4-server --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 ``` -The cache key is the SHA1 of the rendered byte prefix, and files are named -`.kv`. The DS4 payload still stores the exact token IDs and graph state -for that prefix. This matters for continued chats: the model may have generated -one token whose decoded text is later sent back by a client as two canonical -prompt tokens. A rendered byte-prefix hit can still reuse the checkpoint and -tokenize only the new suffix. -The file is intentionally written with ordinary `read`/`write` I/O, not -`mmap`, so restoring cache entries does not add more VM mappings to a process -that already maps the model. +缓存键是已渲染字节前缀的 SHA1，文件命名为 +`.kv`。DS4 负载仍会存储该前缀对应的精确 token ID 与图状态。这对续聊很重要：模型可能曾生成一个 token，其解码文本随后被客户端作为两个规范化 prompt token 回传。命中已渲染字节前缀时仍可复用检查点，且仅对新增后缀进行分词。 +该文件刻意使用普通的 `read`/`write` I/O，而非 `mmap`，因此恢复缓存项不会为已映射模型的进程增加更多 VM 映射。 -Tool calls also keep a bounded exact-DSML replay map keyed by unguessable tool -IDs, so client JSON history can be rendered back to the exact sampled text. The -RAM map keeps up to 100000 IDs by default; tune it with `--tool-memory-max-ids`. -Use `--disable-exact-dsml-tool-replay` to disable this and fall back to -canonical JSON-to-DSML rendering. +工具调用还会维护一个以不可猜测的工具 ID 为键、规模受限的精确 DSML 重放映射，从而可将客户端 JSON 历史渲染回模型当时采样的精确文本。RAM 映射默认最多保留 100000 个 ID；可通过 `--tool-memory-max-ids` 调整。使用 `--disable-exact-dsml-tool-replay` 可禁用该功能并回退到规范的 JSON 到 DSML 渲染。 -On disk, a cache file is: +磁盘上的缓存文件为： ```text KVC fixed header, 48 bytes @@ -1034,7 +748,7 @@ DS4 session payload, payload_bytes from the KVC header optional tool-id map section ``` -The fixed header is little-endian: +固定头为小端序： ```text 0 u8[3] magic = "KVC" @@ -1052,22 +766,11 @@ The fixed header is little-endian: 40 u64 DS4 session payload byte count ``` -The rendered text is the tokenizer-decoded text for the cached token prefix. -It is both the human-inspectable prefix and the lookup identity: its SHA1 is -the filename, and a file is reusable only when those bytes are a prefix of the -incoming rendered prompt. After load, the exact checkpoint tokens from the DS4 -payload remain authoritative, and only the incoming text suffix after the cached -bytes is tokenized. +已渲染文本即缓存 token 前缀经分词器解码后的文本。它既是可供人工检查的前缀，也是查找标识：其 SHA1 即文件名；仅当这些字节构成传入已渲染 prompt 的前缀时，该文件才可复用。加载后，DS4 负载中的精确检查点 token 仍为权威数据；仅对缓存字节之后的传入文本后缀进行分词。 -The optional tool-id map is present only when header extension bit 0 is set. -Appended sections use fixed bit order, so future extension bits can add fields -without ambiguity. The map stores unguessable API tool call IDs back to the -exact DSML block the model sampled. Only mappings whose DSML block is present -in the rendered cached text are stored. This lets restarted servers render -later client history byte-for-byte like the original model output, even if the -client reorders JSON arguments. +可选的工具 ID 映射仅在头扩展位 0 置位时出现。追加段采用固定位序，以便未来扩展位可无歧义地添加字段。该映射将不可猜测的 API 工具调用 ID 关联回模型采样的精确 DSML 块。仅保存其 DSML 块出现在已渲染缓存文本中的映射。这使重启后的服务器能像原始模型输出一样逐字节渲染后续客户端历史，即便客户端重排了 JSON 参数。 -The current tool-id map section is: +当前工具 ID 映射段为： ```text 0 u8[3] magic = "KTM" @@ -1081,14 +784,9 @@ For each entry: ... bytes exact sampled DSML block ``` -The section is auxiliary replay memory, not model state. A cache hit restores -the session payload first, then loads the map if present. Before rendering a -request, the server can also scan cache files for the tool IDs present in the -client history and load just those mappings, so an exact DSML replay can survive -server restarts even when the matching KV snapshot is not the one ultimately -used for the rendered-prefix hit. +该段为辅助重放内存，而非模型状态。缓存命中时先恢复会话负载，再按需加载映射。渲染请求前，服务器也可扫描缓存文件中客户端历史出现的工具 ID，并仅加载对应映射，从而使精确 DSML 重放能在服务器重启后仍可用，即便匹配到的 KV 快照并非最终用于已渲染前缀命中的那份。 -The DS4 session payload starts with thirteen little-endian `u32` fields: +DS4 会话负载以十三个小端序 `u32` 字段开头： ```text 0 magic = "DSV4" @@ -1106,54 +804,32 @@ The DS4 session payload starts with thirteen little-endian `u32` fields: 12 live raw rows serialized below ``` -Then it stores: +随后存储： -- `u32[token_count]` checkpoint token IDs. -- `float32[vocab_size]` logits for the next token after that checkpoint. -- `u32[layer_count]` compressed attention row counts. -- `u32[layer_count]` ratio-4 indexer row counts. -- For every layer: the live raw sliding-window KV rows, written in logical - position order rather than physical ring order. -- For compressed layers: live compressed KV rows and compressor frontier - tensors. -- For ratio-4 compressed layers: live indexer compressed rows and indexer - frontier tensors. +- `u32[token_count]` 检查点 token ID。 +- `float32[vocab_size]` 该检查点之后下一 token 的 logits。 +- `u32[layer_count]` 压缩注意力行计数。 +- `u32[layer_count]` ratio-4 索引器行计数。 +- 每一层：活跃的原始滑动窗口 KV 行，按逻辑位置顺序而非物理环形顺序写入。 +- 压缩层：活跃压缩 KV 行与压缩器前沿张量。 +- ratio-4 压缩层：活跃索引器压缩行与索引器前沿张量。 -The logits are raw IEEE-754 `float32` values from the host `ds4_session` -buffer. They are saved immediately after the checkpoint tokens so a loaded -snapshot can sample or continue from the exact next-token distribution without -running one extra decode step. MTP draft logits/state are not persisted; after -loading a disk checkpoint the draft state is invalidated and rebuilt by normal -generation. +logits 为来自主机 `ds4_session` 缓冲区的原始 IEEE-754 `float32` 值。它们在检查点 token 之后立即保存，以便加载的快照无需额外一次解码步即可从精确的下一 token 分布采样或继续生成。MTP 草稿 logits/状态不会持久化；加载磁盘检查点后，草稿状态会失效并由常规定向重建。 -Distributed coordinator sessions use the same `DSV4` payload. Worker-owned -layer tensors are pulled during save and merged into the normal layer-ordered -tensor stream; during load the coordinator splits that stream into the current -route and pushes the relevant layer tensors back to the workers. The saved file -does not retain the distributed topology. +分布式协调器会话使用相同的 `DSV4` 负载。保存时拉取各 worker 拥有的层张量并合并到按层序排列的常规张量流；加载时协调器将该流拆分为当前路由，并把相关层张量推回各 worker。保存的文件不保留分布式拓扑。 -The tensor payload is DS4-specific KV/session state, not a generic inference -graph dump. It is expected to be portable only across compatible `ds4.c` -builds for this model layout. +张量负载是 DS4 专用的 KV/会话状态，而非通用推理图转储。预期仅能在与本模型布局兼容的 `ds4.c` 构建之间移植。 -The cache stores checkpoints at four moments: +缓存在四个时刻存储检查点： -- `cold`: after a long first prompt reaches a stable prefix, before generation. -- `continued`: when prefill or generation reaches the next absolute aligned frontier. -- `evict`: before an unrelated request replaces the live in-memory session. -- `shutdown`: when the server exits cleanly. +- `cold`：较长首条 prompt 达到稳定前缀之后、生成开始之前。 +- `continued`：prefill 或生成到达下一个绝对对齐前沿时。 +- `evict`：无关请求将活跃内存中会话替换之前。 +- `shutdown`：服务器干净退出时。 -Cold saves intentionally trim a small token suffix and align down to a prefill -chunk boundary. This avoids common BPE boundary retokenization misses when a -future request appends text to the same prompt. The defaults are conservative: -store prefixes of at least 512 tokens, cold-save prompts up to 30000 tokens, -trim 32 tail tokens, and align to 2048-token chunks. The important knobs are: +冷保存会刻意裁掉一小段 token 后缀，并向下对齐到 prefill 块边界。这可避免未来请求向同一 prompt 追加文本时常见的 BPE 边界重分词失效。默认值较保守：至少存储 512 token 的前缀，冷保存的 prompt 上限 30000 token，裁掉尾部 32 个 token，并对齐到 2048 token 块。重要可调项为： -Continued saves use the same alignment and are written only when the live graph -naturally reaches an absolute frontier. With the defaults this means roughly -every 10k tokens, independent of where the first cold checkpoint landed, so long -generations leave restart points behind without persisting the fragile final few -tokens. +持续保存采用相同对齐方式，且仅在活跃图自然到达绝对前沿时才写入。使用默认值时，这意味着大约每 10k token 一次，与首个冷检查点落在何处无关，因此长生成会在不持久化脆弱末尾 token 的情况下留下重启点。 - `--kv-cache-min-tokens` - `--kv-cache-cold-max-tokens` @@ -1163,35 +839,27 @@ tokens. - `--tool-memory-max-ids` - `--disable-exact-dsml-tool-replay` -By default, checkpoints may be reused across the 2-bit and 4-bit routed-expert -variants if the rendered prefix matches. Use `--kv-cache-reject-different-quant` -when you want strict same-quant reuse only. +默认情况下，若已渲染前缀匹配，检查点可在 2-bit 与 4-bit routed-expert 变体间复用。若只想严格同量化复用，请使用 `--kv-cache-reject-different-quant`。 -The cache directory is disposable. If behavior looks suspicious, stop the -server and remove it. You can investigate what is cached with hexdump as -the kv cache files include the verbatim prompt cached. +缓存目录可随时删除。若行为可疑，停止服务器并移除该目录即可。可用 hexdump 查看缓存内容，因为 kv 缓存文件包含逐字缓存的 prompt。 -## Backends +## 后端 -The default graph backend is Metal on macOS and CUDA in CUDA builds: +macOS 上默认图后端为 Metal，CUDA 构建中则为 CUDA： ```sh ./ds4 -p "Hello" --metal ./ds4 -p "Hello" --cuda ``` -On Linux, plain `make` prints the available build targets instead of selecting a -CUDA target implicitly. Use `make cuda-spark` for DGX Spark / GB10. It omits an -explicit `nvcc -arch` because that is currently the fastest path on GB10. Use -`make cuda-generic` for a normal local CUDA build, or set `CUDA_ARCH` explicitly -when cross-building or when you need a known target: +在 Linux 上，普通 `make` 会打印可用构建目标，而非隐式选择 CUDA 目标。DGX Spark / GB10 请使用 `make cuda-spark`。它省略显式 `nvcc -arch`，因为这在 GB10 上目前是最快路径。常规本地 CUDA 构建请使用 `make cuda-generic`；交叉编译或需要已知目标时请显式设置 `CUDA_ARCH`： ```sh make cuda CUDA_ARCH=sm_120 make cuda CUDA_ARCH=native ``` -There is also a CPU reference/debug path: +还提供 CPU 参考/调试路径： ```sh ./ds4 -p "Hello" --cpu @@ -1200,35 +868,20 @@ make cpu ./ds4 -p "Hello" ``` -Do not treat the CPU path as the production target. The CLI and `ds4-server` -support the CPU backend for reference/debug use and share the same KV session -and snapshot format as Metal and CUDA, but normal inference should use Metal or -CUDA. +不要将 CPU 路径视为生产目标。CLI 与 `ds4-server` 支持 CPU 后端用于参考/调试，并与 Metal、CUDA 共享相同的 KV 会话与快照格式，但常规定向应使用 Metal 或 CUDA。 -## Steering +## 引导（Steering） -This project supports steering with single-vector activation directions; see the -`dir-steering` directory for more information. This follows the core idea of the +本项目支持以单向量激活方向进行引导；更多信息见 `dir-steering` 目录。这遵循 [Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717) -paper. You can use it to make the model more or less verbose, less likely to -answer programming questions if it is a chatbot for your car rental web site, -and so forth, much faster than fine-tuning. -This is also useful for cybersecurity researchers who want to reduce a model's -willingness to provide dual-use or offensive security guidance. +论文的核心思想。你可借此让模型更啰嗦或更简洁、降低其回答编程问题的倾向（例如汽车租赁网站的聊天机器人），等等，且比微调快得多。 +这对希望降低模型提供两用或攻击性安全指导意愿的网络安全研究者也很有用。 ## Test Vectors -`tests/test-vectors` contains short and long-context continuation vectors -captured from the official DeepSeek V4 Flash API. The requests use -`deepseek-v4-flash`, greedy decoding, thinking disabled, and the maximum -`top_logprobs` slice exposed by the API. Local vectors are generated with -`./ds4 --dump-logprobs` and compared by token bytes, so tokenizer/template or -attention regressions show up before they become long generation failures. The -C runner pins `DS4_METAL_PREFILL_CHUNK=2048` for this strict API-vector -comparison. +`tests/test-vectors` 包含从官方 DeepSeek V4 Flash API 捕获的短上下文与长上下文延续向量（continuation vectors）。请求使用 `deepseek-v4-flash`、贪心解码（greedy decoding）、禁用思考（thinking disabled），以及 API 所暴露的最大 `top_logprobs` 切片。本地向量由 `./ds4 --dump-logprobs` 生成，并按 token 字节进行比较，从而在 tokenizer/模板或 attention 回归（regressions）演变为长时间生成失败之前就能发现它们。C runner 为此严格 API 向量比对固定使用 `DS4_METAL_PREFILL_CHUNK=2048`。 -All project tests are driven by the C runner, with a small `ds4-eval` -extractor self-test run first: +所有项目测试均由 C runner 驱动，并首先运行一个小型的 `ds4-eval` 提取器自检： ```sh make test # ./ds4-eval --self-test-extractors && ./ds4_test --all @@ -1238,8 +891,7 @@ make test # ./ds4-eval --self-test-extractors && ./ds4_test --a ## Debugging Notes -When a generation looks wrong, three small tools are usually enough to get a -first answer: +当生成结果看起来不对时，通常三个小工具就足以给出初步答案： ```sh ./ds4 --dump-tokens -p "..." @@ -1248,12 +900,6 @@ first answer: ./ds4-server --trace /tmp/ds4-trace.txt ... ``` -- `--dump-tokens` tokenizes the `-p` or `--prompt-file` string exactly as - written, recognizes DS4 protocol specials, and then exits before inference - starts. For example, the DSML tool close marker starts as two tokens: `