From e415cda245eb082e4dd2bfb5500f7072298cf8bf Mon Sep 17 00:00:00 2001 From: wehub-resource-sync Date: Mon, 13 Jul 2026 10:24:27 +0000 Subject: [PATCH] docs: make Chinese README the default --- README.md | 276 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 141 insertions(+), 135 deletions(-) diff --git a/README.md b/README.md index 9211e35..a4988f1 100644 --- a/README.md +++ b/README.md @@ -1,48 +1,54 @@ + +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/Mininglamp-AI/cider) · [上游 README](https://github.com/Mininglamp-AI/cider/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 + # cider -Cider is developed on top of MLX for macOS. It provides: +Cider 基于 MLX 为 macOS 开发。它提供： -1. **Online activation quantization** (W8A8/W4A8) absent in MLX, with custom INT8 TensorOps kernels built as MLX custom primitives supporting full lazy evaluation. -2. **Optimized SDPA decode kernel** inspired by [FlashInfer](https://github.com/flashinfer-ai/flashinfer), featuring contiguous chunk GQA scheduling and register tiling for faster autoregressive decode. +1. MLX 所不具备的**在线激活量化**（W8A8/W4A8），配有自定义 INT8 TensorOps 内核，作为 MLX 自定义原语（custom primitives）构建，支持完整的惰性求值（lazy evaluation）。 +2. 受 [FlashInfer](https://github.com/flashinfer-ai/flashinfer), 启发的**优化 SDPA 解码内核**，采用连续块 GQA 调度与寄存器分块（register tiling），以实现更快的自回归解码。 -It also includes service-side extensions and non-intrusive compatibility patches for mlx_vlm (validated on mlx_vlm 0.4.3), including fixes for Qwen3-VL multi-image inference issues related to RoPE position handling and chunked prefill. +它还包含服务端扩展以及对 mlx_vlm 的非侵入式兼容性补丁（已在 mlx_vlm 0.4.3 上验证），包括修复与 RoPE 位置处理及分块预填充（chunked prefill）相关的 Qwen3-VL 多图推理问题。 -## Conditional Compilation (M4 / M5) +## 条件编译（M4 / M5） -Cider uses **conditional compilation**: the INT8 TensorOps C++ extension is only built on Apple M5+. +Cider 采用**条件编译**：INT8 TensorOps C++ 扩展仅在 Apple M5+ 上构建。 | Chip | `pip install -e .` behavior | `import cider` behavior | |------|----------------------------|------------------------| -| **M5+** | Full build (CMake + Metal kernels) | All features available | -| **M4 and below** | Skips C++ build, installs pure-Python package | `is_available()` → False, `convert_model()` is a warning no-op | +| **M5+** | 完整构建（CMake + Metal kernels） | 全部功能可用 | +| **M4 及以下** | 跳过 C++ 构建，安装纯 Python 包 | `is_available()` → False，`convert_model()` 为警告空操作（warning no-op） | -**Override via environment variable:** +**通过环境变量覆盖：** ```bash CIDER_FORCE_BUILD=1 pip install -e . # Force build (e.g., CI) CIDER_FORCE_BUILD=0 pip install -e . # Force skip ``` -## Modes +## 模式 | Mode | Weights | Activations | Compute Path | Status | |------|---------|-------------|--------------|--------| -| **W8A8** | INT8 symmetric | INT8 per-token | TensorOps matmul2d | ✅ Implemented | -| **W4A8** | INT4 packed (uint8) | INT8 per-token | Unpack→TensorOps | ✅ Implemented | -| W4A16 | — | — | MLX built-in | Baseline | -| W8A16 | — | — | MLX built-in | Baseline | +| **W8A8** | INT8 symmetric | INT8 per-token | TensorOps matmul2d | ✅ 已实现 | +| **W4A8** | INT4 packed (uint8) | INT8 per-token | Unpack→TensorOps | ✅ 已实现 | +| W4A16 | — | — | MLX built-in | 基线 | +| W8A16 | — | — | MLX built-in | 基线 | -**W4A16 and W8A16 are already supported by MLX natively** — this SDK provides the missing W8A8 and W4A8 modes that MLX does not implement. +**W4A16 和 W8A16 已由 MLX 原生支持** — 本 SDK 提供 MLX 未实现的 W8A8 与 W4A8 模式。 -## Optimized SDPA (Decode Attention) +## 优化 SDPA（解码注意力） -Cider includes an optimized **Scaled Dot-Product Attention** kernel for autoregressive decode (`Q_seq=1`), inspired by [FlashInfer](https://github.com/flashinfer-ai/flashinfer). Key techniques: +Cider 包含面向自回归解码（`Q_seq=1`）的优化**缩放点积注意力**（Scaled Dot-Product Attention）内核，受 [FlashInfer](https://github.com/flashinfer-ai/flashinfer). 启发。关键技术： -- **Contiguous Chunk GQA Scheduling**: Instead of dispatching each Q head independently (reading KV `H_q` times), a single threadgroup processes all Q heads sharing the same KV group. This reduces DRAM bandwidth by up to `gqa_factor`×. -- **Register Tiling (TILE=4)**: Each thread holds partial results for 4 Q heads in registers — no shared memory, no barrier synchronization — maximizing ALU throughput for grouped-query decode. -- **2-Pass FlashDecoding**: For long sequences (≥1024 tokens), a 2-pass reduction partitions the KV sequence across threadgroups to improve occupancy. -- **AutoTune**: Automatic sweep of block sizes per (architecture, GQA ratio, head count, sequence length), cached to `~/.cider_sdpa_tune.json`. +- **连续块 GQA 调度（Contiguous Chunk GQA Scheduling）**：不再让每个 Q 头独立派发（读取 KV `H_q` 次），而是由单个线程组处理共享同一 KV 组的所有 Q 头。这可减少最多 `gqa_factor`× 的 DRAM 带宽。 +- **寄存器分块（Register Tiling，TILE=4）**：每个线程在寄存器中保存 4 个 Q 头的部分结果 — 无需共享内存，无需屏障同步 — 最大化分组查询解码的 ALU 吞吐。 +- **两遍 FlashDecoding（2-Pass FlashDecoding）**：对于长序列（≥1024 tokens），两遍归约将 KV 序列划分到各线程组，以提升占用率。 +- **AutoTune**：针对（架构、GQA 比率、头数、序列长度）自动扫描块大小，并缓存至 `~/.cider_sdpa_tune.json`。 -### Usage +### 用法 ```python import cider @@ -57,11 +63,11 @@ cider.autotune_sdpa() cider.unpatch_sdpa() ``` -The patch is **fully transparent**: prefill (`Q_seq > 1`) and masked attention automatically fall back to MLX's native implementation. Only decode steps (`Q_seq=1`, no mask) route through Cider's optimized kernel. **Zero code changes** to mlx_lm or mlx_vlm. +该补丁**完全透明**：预填充（`Q_seq > 1`）与带掩码注意力会自动回退到 MLX 原生实现。仅解码步骤（`Q_seq=1`，无掩码）会走 Cider 的优化内核。**无需修改** mlx_lm 或 mlx_vlm 代码。 -### SDPA Micro-Benchmark (Apple M5 Pro, D=128) +### SDPA 微基准（Apple M5 Pro，D=128） -Speedup vs MLX `mx.fast.scaled_dot_product_attention`: +相对 MLX `mx.fast.scaled_dot_product_attention` 的加速比： | Config | N=1K | N=2K | N=4K | N=8K | N=16K | N=32K | |--------|------|------|------|------|-------|-------| @@ -69,11 +75,11 @@ Speedup vs MLX `mx.fast.scaled_dot_product_attention`: | GQA4 (H_q=32, H_kv=8) | 1.04x | 1.06x | 1.06x | 1.12x | 1.16x | 1.21x | | GQA8 (H_q=32, H_kv=4) | 1.04x | 1.07x | 1.13x | 1.21x | 1.34x | **1.57x** | -Speedup range: **0.99×–1.57×**. Correctness: 18/18 PASS (max diff ≤ 0.000122 vs MLX, fp16). 14 WIN / 4 TIE / 0 LOSS — zero regressions. +加速范围：**0.99×–1.57×**。正确性：18/18 PASS（相对 MLX 最大差异 ≤ 0.000122，fp16）。14 WIN / 4 TIE / 0 LOSS — 零回退。 -### End-to-End Decode Speed with SDPA (Qwen3-VL-4B, Apple M5 Pro) +### 启用 SDPA 的端到端解码速度（Qwen3-VL-4B，Apple M5 Pro） -Tested with `benchmarks/bench_e2e_wxa16.py` on real multi-image VLM trajectories (1–3 screenshots, 1334–3455 prompt tokens): +使用 `benchmarks/bench_e2e_wxa16.py` 在真实多图 VLM 轨迹上测试（1–3 张截图，1334–3455 prompt tokens）： | Config | Decode tok/s | vs Baseline | |--------|:-----------:|:-----------:| @@ -86,30 +92,30 @@ Tested with `benchmarks/bench_e2e_wxa16.py` on real multi-image VLM trajectories | W8A8 per-channel | 49.1 | — | | W8A8 per-channel + Cider SDPA | **52.6** | **+7.1%** | -Cider SDPA stacks on top of W8A8 prefill gains. The benefit scales with GQA ratio and KV cache length — models with higher GQA ratios (e.g., Qwen3-32B with GQA8) and longer contexts see larger gains. +Cider SDPA 可叠加在 W8A8 预填充收益之上。收益随 GQA 比率与 KV cache 长度而放大 — GQA 比率更高（例如 GQA8 的 Qwen3-32B）且上下文更长的模型获益更大。 -MLX's quantization is **weight-only**: QuantizedLinear dequantizes weights to FP16 and uses FP16 GEMM. While MLX's Steel NAX templates are generic enough to be instantiated with INT8 types (and would achieve identical raw matmul throughput — [see our transparent benchmark](benchmarks/mlx_native/cider_vs_mlx_int8.md)), MLX does not provide the quantization/dequantization pipeline needed for actual W8A8 inference. Cider fills this gap with fused quantize-matmul-dequant primitives. +MLX 的量化是**仅权重量化（weight-only）**：QuantizedLinear 将权重反量化为 FP16 并使用 FP16 GEMM。虽然 MLX 的 Steel NAX 模板足够通用，可实例化为 INT8 类型（且能达到相同的原始 matmul 吞吐 — [参见我们的透明基准测试](benchmarks/mlx_native/cider_vs_mlx_int8.md)），但 MLX 并未提供实际 W8A8 推理所需的量化/反量化流水线。Cider 通过融合的 quantize-matmul-dequant 原语填补这一缺口。 -This SDK implements online INT8 activation quantization and INT8 TensorOps-based compute for the supported inference paths. +本 SDK 为受支持的推理路径实现在线 INT8 激活量化，以及基于 INT8 TensorOps 的计算。 -### W8A8 Quantization Granularity +### W8A8 量化粒度 | Granularity | Description | Speed | Precision | |-------------|-------------|-------|-----------| -| **Per-channel** | One scale per output channel | Fastest (1.8x prefill) | Slightly lower | -| **Per-group (gs=128)** | One scale per 128 elements | Fast (1.5x prefill) | Moderate precision retention | -| **Per-group (gs=64)** | One scale per 64 elements | Moderate (1.3x prefill) | Higher precision | +| **Per-channel** | 每个输出通道一个 scale | 最快（预填充 1.8x） | 略低 | +| **Per-group (gs=128)** | 每 128 个元素一个 scale | 较快（预填充 1.5x） | 精度保留适中 | +| **Per-group (gs=64)** | 每 64 个元素一个 scale | 中等（预填充 1.3x） | 精度更高 | -## Performance (Apple M5 Pro) +## 性能（Apple M5 Pro） -### Individual Operator Latency +### 单算子延迟 Shape [N=10240, K=2560] | M | PC(ms) | PG(ms) | w8a16 | w4a16 | PC/w8 | PC/w4 | PG/w8 | PG/w4| |-----|------|------|-----|-----|-----|------|-----|----| | 1 | 0.27ms | 0.26ms | 0.26ms | 0.18ms | 0.96x | 0.67x | 0.99x | 0.69x | |128 | 0.34ms | 0.39ms | 0.49ms | 0.44ms | 1.43x | 1.28x | 1.26x | 1.13x | -|1024 | 1.23ms | 1.52ms | 2.24ms | 2.04ms | 1.82x | 1.66x | 1.47x | 1.34x| +|1024 | 1.23ms | 1.52ms | 2.24ms | 2.04ms | 1.82x | 1.66x | 1.47x | 1.34x| |4096 | 4.41ms | 5.65ms | 8.12ms | 7.72ms | 1.84x | 1.75x | 1.44x | 1.37x | |8192 | 8.71ms | 11.40ms | 16.23ms | 15.09ms | 1.86x | 1.73x | 1.42x | 1.32x| @@ -123,7 +129,7 @@ Shape [N=2560, K=10240] | 4096 | 5.37ms | 6.79ms | 8.54ms | 8.04ms | 1.59x | 1.50x | 1.26x | 1.18x | | 8192 | 10.97ms | 12.94ms | 17.28ms | 16.23ms | 1.58x | 1.48x | 1.34x | 1.25x | -### End-to-End VLM +### 端到端 VLM **Qwen3-VL-2B** @@ -142,7 +148,7 @@ Shape [N=2560, K=10240] | 3455 | 1755 | 1603 | **1881** | 30 | **52** | 49 | -### LLM Quantization: Precision vs. Speed Comparison +### LLM 量化：精度与速度对比 @@ -226,31 +232,31 @@ Shape [N=2560, K=10240]
-## NOTE -It's important to note that cider assumes the weights are **quantization-friendly** . This means you need to ensure your model has already undergone some post-training quantization calibration methods, such as GTPQ, SmoothQuant, or Quarot etc, to handle outliers or the model itself was trained using QAT. When cider transforms the model (e.g., with w8a8) and calculates the quantization scale, it defaults to the simplest **min-max** method, although it also supports the 99th percentile method (clip_percentile=99.9). If you encounter garbled output, it's highly likely that your model is affected by **outliers** . Therefore, some preprocessing is **necessary**. +## 说明 +需要注意的是，cider 假定权重是 **量化友好（quantization-friendly）** 的。这意味着你需要确保模型已经经过某种训练后量化校准方法，例如 GTPQ、SmoothQuant 或 Quarot 等，以处理异常值（outliers），或者模型本身是通过 QAT（量化感知训练）训练的。当 cider 转换模型（例如使用 w8a8）并计算量化 scale 时，默认采用最简单的 **min-max** 方法，同时也支持第 99 百分位方法（clip_percentile=99.9）。如果遇到乱码输出，很大概率是模型受到了 **异常值（outliers）** 的影响。因此，某些预处理步骤是 **必要的**。 -## Requirements +## 环境要求 -- **Apple M5+** for INT8 TensorOps (M4 and below: installs as pure-Python, `is_available()` returns False) +- **Apple M5+** 用于 INT8 TensorOps（M4 及以下：以纯 Python 方式安装，`is_available()` 返回 False） - Python 3.12+ - MLX >= 0.31 -- nanobind >= 2.12 (only needed on M5+ for C++ build) -- CMake >= 3.27 (only needed on M5+ for C++ build) +- nanobind >= 2.12（仅在 M5+ 上进行 C++ 编译时需要） +- CMake >= 3.27（仅在 M5+ 上进行 C++ 编译时需要） -## Install +## 安装 ```bash pip install -e . ``` -On M5+, this runs CMake to compile the C++ extension, then installs the Python package. -On M4 and below, only the Python package is installed (no compilation errors). +在 M5+ 上，这会运行 CMake 编译 C++ 扩展，然后安装 Python 包。 +在 M4 及以下设备上，仅安装 Python 包（不会出现编译错误）。 -## Quick Start +## 快速开始 -### One-line Model Conversion (Recommended) +### 一行命令完成模型转换（推荐） ```python from cider import convert_model, is_available @@ -266,13 +272,13 @@ else: pass # Falls back to standard MLX inference on M4 ``` -**Important** -When quantizing Vision-Language Models (VLMs), the vision transformer (ViT) is generally not replaced. Directly using convert_model will quantize the vision model's linear layers as well, which typically causes accuracy drop. For VLMs, we recommend calling convert_model(model.language_model) to apply existing quantization methods like GPTQ, SmoothQuant, and AWQ to the language model only. +**重要提示** +在量化视觉语言模型（Vision-Language Models，VLMs）时，视觉 Transformer（ViT）通常不会被替换。直接使用 convert_model 也会量化视觉模型的线性层，这通常会导致精度下降。对于 VLMs，我们建议调用 convert_model(model.language_model)，仅对语言模型应用 GPTQ、SmoothQuant 和 AWQ 等现有量化方法。 -Tested on selected MLX transformer models, including Qwen3, Qwen3-VL and Llama3 families. Other architectures may require adaptation. +已在选定的 MLX Transformer 模型上测试，包括 Qwen3、Qwen3-VL 和 Llama3 系列。其他架构可能需要适配。 -### Layer-level API +### 层级别 API ```python import numpy as np @@ -301,7 +307,7 @@ y4 = layer4(x) mx.eval(y4) ``` -## Low-Level API +## 底层 API ```python from cider import perchannel_linear, w4a8_linear, quantize_weight_int8, pack_weight_int4 @@ -315,7 +321,7 @@ y = perchannel_linear(x, mx.array(w_int8), mx.array(scale)) y4 = w4a8_linear(x, mx.array(packed_w4), mx.array(scale4)) ``` -## Project Structure +## 项目结构 ``` cider/ @@ -383,13 +389,13 @@ cider/ └── README.md ``` -## VLM Inference Service +## VLM 推理服务 -`vlm_service/` provides a ready-to-use **OpenAI-style** VLM inference server with W8A8 acceleration. +`vlm_service/` 提供了一个开箱即用的 **OpenAI 风格** VLM 推理服务器，支持 W8A8 加速。 -### Quick Start +### 快速开始 -1. **Configure** `config/config.yaml`: +1. **配置** `config/config.yaml`： ```yaml model_name_or_path: /path/to/your/model # MLX VLM model (e.g., Qwen3-VL-2B W8A16) @@ -405,18 +411,18 @@ w8a8: mode: 'off' # 'auto' | 'on' | 'off' ``` -- `auto`: Enable W8A8 if hardware supports it, fallback to default otherwise -- `on`: Force W8A8 (error if unsupported). "When 'on' is selected, it means your model needs to perform online activation quantization. In this case, Cider itself does **not** guarantee quantization accuracy, and you need to apply some quantization algorithms yourself, such as SmoothQuant, QuaRot, GPTQ, or even QAT, to ensure that the accuracy does not degrade significantly after activation quantization. This option simply provides a way for you to leverage the hardware's computational advantages when your model applies W8A8, rather than just simulating quantization." -- `off`: Disable W8A8, use standard MLX inference +- `auto`：若硬件支持则启用 W8A8，否则回退到默认设置 +- `on`：强制启用 W8A8（不支持时报错）。"选择 'on' 时，表示你的模型需要执行在线激活量化（online activation quantization）。在这种情况下，Cider 本身**不**保证量化精度，你需要自行应用某些量化算法，例如 SmoothQuant、QuaRot、GPTQ 或 QAT，以确保激活量化后精度不会明显下降。此选项只是让你在模型采用 W8A8 时能够利用硬件的计算优势，而不仅仅是模拟量化。" +- `off`：禁用 W8A8，使用标准 MLX 推理 -2. **Start the server**: +2. **启动服务器**： ```bash cd vlm_service python server.py --config ../config/config.yaml ``` -3. **Send requests** (OpenAI-style API): +3. **发送请求**（OpenAI 风格 API）： ```bash # Text-only @@ -441,95 +447,95 @@ curl http://localhost:8341/v1/chat/completions \ }' ``` -### API Endpoints +### API 端点 | Endpoint | Method | Description | |----------|--------|-------------| -| `/v1/chat/completions` | POST | Chat completion (stream / non-stream) | -| `/v1/models` | GET | List available models | -| `/health` | GET | Health check | -| `/v1/queue` | GET | Request queue status | +| `/v1/chat/completions` | POST | 对话补全（流式 / 非流式） | +| `/v1/models` | GET | 列出可用模型 | +| `/health` | GET | 健康检查 | +| `/v1/queue` | GET | 请求队列状态 | -### How W8A8 Works in the Service +### 服务中 W8A8 的工作方式 -When `w8a8.mode` is `auto` or `on`, the server calls `cider.convert_model()` at startup to replace all Linear layers with `CiderLinear`. During inference: +当 `w8a8.mode` 为 `auto` 或 `on` 时，服务器在启动时会调用 `cider.convert_model()`，将所有 Linear 层替换为 `CiderLinear`。推理过程中： -- **Prefill** (processing input tokens, seq_len > 1): Uses W8A8 INT8 GEMM for faster computation -- **Decode** (generating tokens one by one, seq_len == 1): Uses INT8 MV kernel (near-native speed) +- **Prefill**（处理输入 token，seq_len > 1）：使用 W8A8 INT8 GEMM 以加速计算 +- **Decode**（逐 token 生成，seq_len == 1）：使用 INT8 MV kernel（接近原生速度） -No code changes needed — the switching is automatic based on input sequence length. +无需修改代码——切换会根据输入序列长度自动进行。 -## Architecture +## 架构 -### MLX Custom Primitives +### MLX 自定义原语 -Both W8A8Linear and W4A8Linear are implemented as `mlx::core::Primitive` subclasses. This means: +W8A8Linear 与 W4A8Linear 均实现为 `mlx::core::Primitive` 的子类。这意味着： -1. **Lazy evaluation**: `y = layer(x)` builds a graph node, not immediate computation -2. **Graph composition**: Multiple primitive calls compose into a single MLX graph -3. **Stream scheduling**: MLX's scheduler handles GPU dispatch order +1. **惰性求值（lazy evaluation）**：`y = layer(x)` 构建图节点，而非立即计算 +2. **图组合**：多个原语调用组合为单个 MLX 图 +3. **流调度**：MLX 的调度器处理 GPU 派发顺序 -### Metal Kernel Pipeline +### Metal Kernel 流水线 -Each primitive dispatches Metal compute kernels: +每个原语都会派发 Metal 计算 kernel： -**Prefill (M > 1):** -1. **quantize_per_token**: FP16 activations → INT8 + per-token scales -2. **matmul_fused_dequant**: INT8 × INT8 → INT32 → FP16 (with fused scale dequantization) +**Prefill（M > 1）：** +1. **quantize_per_token**：FP16 激活 → INT8 + 逐 token 缩放 +2. **matmul_fused_dequant**：INT8 × INT8 → INT32 → FP16（融合缩放反量化） -**Decode (M = 1):** -- **int8_mv**: Direct INT8 matrix-vector product with on-the-fly weight dequantization (no activation quantization needed) +**Decode（M = 1）：** +- **int8_mv**：直接 INT8 矩阵-向量乘积，权重反量化即时进行（无需激活量化） -For W4A8, the GEMM step includes inline INT4→INT8 unpacking in the fragment load. +对于 W4A8，GEMM 步骤在 fragment load 中包含内联 INT4→INT8 解包。 ### TensorOps matmul2d -The INT8 GEMM uses Apple's `mpp::tensor_ops::matmul2d(16, 32, 16)` — hardware-accelerated INT8×INT8→INT32 matrix multiply available on M5+ via Metal 4's `cooperative_tensor` API. This is the same hardware instruction available to MLX's NAX templates. Cider's kernel adds fused dequantization (INT32 × scales → FP16) in the store phase, avoiding an extra device memory round-trip. See [kernel comparison](benchmarks/mlx_native/cider_vs_mlx_int8.md) for details. +INT8 GEMM 使用 Apple 的 `mpp::tensor_ops::matmul2d(16, 32, 16)`——通过 Metal 4 的 `cooperative_tensor` API，在 M5+ 上可用的硬件加速 INT8×INT8→INT32 矩阵乘法。这与 MLX 的 NAX 模板可用的硬件指令相同。Cider 的 kernel 在存储阶段加入融合反量化（INT32 × scales → FP16），避免额外的设备内存往返。详见 [kernel comparison](benchmarks/mlx_native/cider_vs_mlx_int8.md)。 -### Tile Configurations +### Tile 配置 | Config | BM | BN | BK | SK | Threads | Use When | |--------|----|----|----|----|---------|----------| | Large | 128 | 128 | 512 | 32 | 512 | M > 64 | | Small | 32 | 128 | 512 | 32 | 128 | M ≤ 64 | -Auto-selected based on M. L2 cache swizzle dispatch included. +根据 M 自动选择。包含 L2 cache swizzle 派发。 -## ANE+GPU Heterogeneous Tensor Parallelism (experimental) +## ANE+GPU 异构张量并行（实验性） -We found that during inference on Mac, only two hardware computing units—GPU and CPU—were utilized, while the ANE (Apple Neural Engine) computing unit on Mac remained idle. We identified this as a potential optimization opportunity. Inspired by [maderix/ANE](https://github.com/maderix/ANE), we conducted experimental work on a hybrid ANE+GPU inference mode. Currently, we apply this approach to tensor parallel computing. On the M4 chip, during synchronous-only forward inference (MLX natively uses a technique called lazy evaluation, which reduces synchronization overhead; in end-to-end testing, the hybrid inference currently shows no advantage, mainly because we have not yet implemented this using MLX's lazy evaluation—this remains future work), we observed approximately **3%~16%** performance improvement compared to pure GPU inference under synchronize pipeline. This remains exploratory work, and end-to-end gains are currently limited by the lack of a lazy-evaluation-compatible implementation. +我们发现，在 Mac 上进行推理时，仅利用了 GPU 与 CPU 两个硬件计算单元，而 Mac 上的 ANE（Apple Neural Engine）计算单元处于空闲状态。我们将此视为潜在的优化机会。受 [maderix/ANE](https://github.com/maderix/ANE), 启发，我们开展了 ANE+GPU 混合推理模式的实验工作。目前我们将该方案应用于张量并行计算。在 M4 芯片上，在仅同步的前向推理期间（MLX 原生使用称为惰性求值的技术以降低同步开销；在端到端测试中，混合推理目前尚未显示出优势，主要因为我们尚未基于 MLX 的惰性求值实现该方案——这仍是后续工作），与纯 GPU 推理在同步流水线（synchronize pipeline）下相比，我们观察到约 **3%~16%** 的性能提升。这仍属探索性工作，端到端收益目前受限于缺少与惰性求值兼容的实现。 -During LLM prefill, the GPU's matrix units are fully occupied — but the **Apple Neural Engine sits completely idle**. ANE Split exploits this by splitting each linear layer's GEMM along output channels: +在 LLM prefill 阶段，GPU 的矩阵单元已满载——但 **Apple Neural Engine 完全空闲**。ANE Split 利用这一点，沿输出通道拆分每个线性层的 GEMM： -- **ANE** computes ~65% of output channels (FP32, via reverse-engineered private `_ANEClient` API) -- **GPU** computes the remaining ~35% (FP16, standard MLX matmul) -- Both run **concurrently**, and results are concatenated +- **ANE** 计算约 65% 的输出通道（FP32，通过逆向工程的私有 `_ANEClient` API） +- **GPU** 计算剩余约 35%（FP16，标准 MLX matmul） +- 两者**并发**运行，结果拼接 -This is a form of **heterogeneous tensor parallelism** — not data parallelism, not pipeline parallelism — exploiting two distinct compute units on the same SoC. +这是一种**异构张量并行**——不是数据并行，也不是流水线并行——利用同一 SoC 上的两个不同计算单元。 -### Performance (Apple M4, Qwen3-VL-2B Prefill) +### 性能（Apple M4，Qwen3-VL-2B Prefill） | seq | W8A16 GPU | SplitLinear | Speedup vs W8A16 | |-----|----------|-----------|-------------| | 512 | 639.9 ms | **615.9 ms** | **1.039×** | | 1024 | 1348.6 ms | **1156.9 ms** | **1.17×** | -In the tested benchmark cases, cosine similarity was close to 1.0 and top-1 token agreement was 100%. +在测试的基准案例中，余弦相似度接近 1.0，top-1 token 一致率为 100%。 -### Key Design Choices +### 关键设计选择 -- **Prefill only**: Decode falls back to original GPU linear (zero overhead) -- **Shared input preparation**: Q/K/V and Gate/Up projections share a single input transpose+numpy copy via `_InputGroup` -- **Auto-routing**: Down projections (IC > 2×OC) stay GPU-only where ANE is inefficient -- **Short-seq bypass**: Sequences < 192 tokens skip splitting (overhead > benefit) +- **仅 Prefill**：Decode 回退到原始 GPU linear（零开销） +- **共享输入准备**：Q/K/V 与 Gate/Up 投影通过 `_InputGroup` 共享单次输入转置+numpy 拷贝 +- **自动路由**：Down 投影（IC > 2×OC）在 ANE 效率低下时保持仅 GPU +- **短序列绕过**：序列 < 192 token 时跳过分拆（开销大于收益） -See [`experimental/README.md`](experimental/README.md) for full documentation, usage, and build instructions and limitations. +完整文档、用法、构建说明与限制见 [`experimental/README.md`](experimental/README.md)。 -> **Note:** ANE Split is tested on M4. M5 introduced ANE architecture changes that may break the private API bridge — not yet validated on M5. +> **注意：** ANE Split 已在 M4 上测试。M5 引入了 ANE 架构变更，可能破坏私有 API 桥接——尚未在 M5 上验证。 -## Quantization +## 量化 | Component | Scheme | Granularity | |-----------|--------|-------------| @@ -539,15 +545,15 @@ See [`experimental/README.md`](experimental/README.md) for full documentation, u | Accumulation | INT32 | — | | Output dequant | `C_fp16 = C_int32 * s_act * s_weight` | Per-element | -## Limitations +## 限制 -- **M=1 individual operator**: Per-channel MV kernel is slower than MLX W4A16 for isolated decode calls. The per-group MV kernel is within 5% of MLX W8A16 decode speed in end-to-end benchmarks. -- **Apple M5+ only** for INT8 TensorOps: M4 and below installs but `is_available()` returns False. -- **W4A8 slower than W8A8**: INT4→INT8 unpack ALU overhead (Metal 4 matmul2d has no native INT4 operand). +- **M=1 单算子**：逐通道 MV kernel 在独立 decode 调用时慢于 MLX W4A16。逐组 MV kernel 在端到端基准中与 MLX W8A16 decode 速度相差在 5% 以内。 +- **INT8 TensorOps 仅支持 Apple M5+**：M4 及以下可安装但 `is_available()` 返回 False。 +- **W4A8 慢于 W8A8**：INT4→INT8 解包 ALU 开销（Metal 4 matmul2d 无原生 INT4 操作数）。 -## Tools +## 工具 -### Unified PPL Evaluation +### 统一 PPL 评估 ```bash # Run all 5 configurations in one script @@ -557,32 +563,32 @@ python tools/eval_ppl_all.py --num-samples 50 # Outputs comparison table at the end ``` -## Roadmap +## 路线图 -- [x] One-line model conversion API (`convert_model`, auto prefill/decode) -- [x] Automatic dtype handling (float16 / bfloat16) -- [x] Per-channel and per-group W8A8 quantization -- [x] Dedicated decode MV kernel (matches native MLX speed) -- [x] Conditional compilation (M4 graceful fallback) -- [x] mlx_vlm and mlx_lm integration examples -- [x] Optimized SDPA decode kernel (FlashInfer-inspired, GQA-aware) -- [x] One-line `patch_sdpa()` for transparent decode acceleration -- [x] SDPA AutoTune (per-architecture block size sweep) -- [ ] ANE primitives lazy evaluation -- [ ] Integrated Pruning Feature -- [ ] KVCache quantization +- [x] 一行式模型转换 API（`convert_model`，自动 prefill/decode） +- [x] 自动 dtype 处理（float16 / bfloat16） +- [x] 逐通道与逐组 W8A8 量化 +- [x] 专用 decode MV kernel（匹配原生 MLX 速度） +- [x] 条件编译（M4 优雅回退） +- [x] mlx_vlm 与 mlx_lm 集成示例 +- [x] 优化的 SDPA decode kernel（受 FlashInfer 启发，支持 GQA） +- [x] 一行式 `patch_sdpa()`，透明加速 decode +- [x] SDPA AutoTune（按架构块大小扫描） +- [ ] ANE 原语惰性求值 +- [ ] 集成剪枝功能 +- [ ] KVCache 量化 -## Authors +## 作者 Multimodal Team, Mininglamp Technology -For bug reports, feature requests, and usage questions, please open an issue in this repository. +如有 bug 报告、功能请求或使用问题，请在本仓库提交 issue。 -## Citation +## 引用 -If you find this work useful, please cite: +若您认为本工作有价值，请引用： ```bibtex @software{wang2026cider, @@ -593,11 +599,11 @@ If you find this work useful, please cite: } ``` -## License +## 许可证 MIT -## Acknowledgments +## 致谢 - [MLX](https://github.com/ml-explore/mlx) by Apple — primitive API, NAXFrag kernel architecture - [FlashInfer](https://github.com/flashinfer-ai/flashinfer) — inspired contiguous chunk GQA scheduling and register tiling for SDPA decode kernel