chore: import upstream snapshot with attribution

2026-07-13 13:30:03 +08:00
commit ec436095dd
1232 changed files with 404407 additions and 0 deletions
@@ -0,0 +1,454 @@
+# Weight Quantization Tools
+
+KT-Kernel provides weight conversion tools for CPU-GPU hybrid inference (e.g., integrating KTransformers with SGLang). Both tools work together to enable heterogeneous expert placement:
+
+- **CPU Weights (`convert_cpu_weights.py`)**: Quantize weights to INT4/INT8 with AMX optimization for CPU-resident "cold" experts
+- **GPU Weights (`convert_gpu_weights.py`)**: Apply GPTQ/RTN quantization (W4A16/W8A16) for GPU-resident "hot" experts
+- **KT Fused Expert LoRA (`convert_kt_to_sglang_adapter.py`)**: Convert KT SFT fused expert LoRA checkpoints into adapter-only SafeTensors directories
+
+---
+
+## KT Fused Expert LoRA Adapter Conversion
+
+KT SFT fused expert LoRA saves MoE expert LoRA tensors in `fused_expert_lora.safetensors` using compact 3D tensors:
+
+```
+layers.{L}.experts.gate_lora_a
+layers.{L}.experts.gate_lora_b
+layers.{L}.experts.up_lora_a
+layers.{L}.experts.up_lora_b
+layers.{L}.experts.down_lora_a
+layers.{L}.experts.down_lora_b
+```
+
+Use `convert_kt_to_sglang_adapter.py` to convert raw KT SFT output into one merged SGLang adapter directory:
+
+```bash
+python scripts/convert_kt_to_sglang_adapter.py /path/to/kt_adapter /path/to/sglang_adapter \
+  --base-model-name-or-path /path/to/base_model \
+  --lora-alpha 16 \
+  --overwrite
+```
+
+Output:
+
+```
+sglang_adapter/
+├── adapter_config.json
+└── adapter_model.safetensors
+```
+
+The converter merges the existing non-expert `adapter_model.safetensors` with expanded expert tensors from `fused_expert_lora.safetensors`. Pass this merged directory to SGLang with:
+
+```bash
+--enable-lora \
+--lora-paths my_lora=/path/to/sglang_adapter
+```
+
+The KTransformers SGLang fork will auto-split the merged adapter internally at server startup. Users do not need to pass separate expert and non-expert adapter paths in the normal workflow.
+
+Optional split outputs for debugging:
+
+```bash
+python scripts/convert_kt_to_sglang_adapter.py /path/to/kt_adapter /path/to/sglang_adapter \
+  --base-model-name-or-path /path/to/base_model \
+  --expert-output-dir /path/to/expert_adapter \
+  --nonexpert-output-dir /path/to/nonexpert_adapter \
+  --overwrite
+```
+
+Existing PEFT prefixes such as `base_model.model.` are stripped to match SGLang's loader. Scaling is not folded into the LoRA B tensors. Runtime scaling remains `lora_alpha / r`; if the input directory has no `adapter_config.json`, pass `--lora-alpha` explicitly.
+
+This script only converts adapter files. Serving compatibility depends on the KTransformers SGLang runtime branch being used.
+
+### Optional Integration Validation
+
+The unit tests use synthetic tensors and run without model files. To validate a real KT adapter directory, set these environment variables:
+
+```bash
+export KT_LORA_ADAPTER_DIR=/path/to/kt_adapter
+export KT_LORA_BASE_MODEL=/path/to/base_model
+export KT_LORA_ALPHA=16  # required only if the input has no adapter_config.json
+```
+
+Then run:
+
+```bash
+python -m pytest kt-kernel/test/per_commit/test_convert_kt_to_sglang_adapter_integration.py -q
+```
+
+To run a large adapter conversion smoke test, also set:
+
+```bash
+export KT_LORA_LARGE_ADAPTER_DIR=/path/to/large_kt_adapter
+```
+
+These integration tests check real fused tensor splitting, optional `adapter_model.safetensors` merging, `adapter_config.json` compatibility with `sglang.srt.lora.lora_config.LoRAConfig`, and large-file readability. They intentionally do not start an SGLang server or validate runtime `FusedMoE` LoRA application.
+
+---
+
+## CPU Weight Quantization
+
+Convert weights to INT4/INT8 format optimized for AMX inference on CPU. These quantized weights are used for "cold" experts (less frequently accessed) that run on CPU in hybrid inference scenarios.
+
+### Quantization Methods
+
+- **INT4**: 4-bit quantization for maximum memory efficiency
+- **INT8**: 8-bit quantization for better accuracy
+
+### Supported Input Formats
+
+- **FP8**: 8-bit floating point with automatic dequantization
+- **FP16**: 16-bit floating point
+- **BF16**: BFloat16 format
+
+> **⚠️ Precision Warning:** Quantizing directly from FP8 to INT4/INT8 may cause significant accuracy degradation. For best results, use the original **BF16** model as the source for INT4/INT8 quantization.
+
+## Basic Usage
+
+### Quantize BF16 model to INT4
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /path/to/bf16/model \
+  --input-type bf16 \
+  --output /path/to/output \
+  --quant-method int4
+```
+
+### Quantize FP16 model to INT8
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /path/to/fp16/model \
+  --input-type fp16 \
+  --output /path/to/output \
+  --quant-method int8
+```
+
+### Quantize FP8 model to INT4
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /path/to/fp8/model \
+  --input-type fp8 \
+  --output /path/to/output \
+  --quant-method int4
+```
+
+## Output Format
+
+By default, the converted weights are saved in SafeTensors format with NUMA-aware layout:
+
+```
+output_dir/
+├── model-00001-of-00050.safetensors
+├── model-00002-of-00050.safetensors
+├── ...
+├── config.json
+└── tokenizer files...
+```
+
+Each expert's weights are split across NUMA nodes for optimal memory access:
+- `blk.{layer}.ffn_{proj}_exps.{expert}.numa.{numa_idx}.weight`: Quantized weights
+- `blk.{layer}.ffn_{proj}_exps.{expert}.numa.{numa_idx}.scale`: Quantization scales
+
+## Advanced Options
+
+### Low Memory Mode
+
+For systems with insufficient memory to complete full model quantization, use the `--no-merge-safetensor` flag to keep weights in layer folder structure without merging into safetensor files:
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /path/to/model \
+  --input-type bf16 \
+  --output /path/to/output \
+  --quant-method int4 \
+  --no-merge-safetensor
+```
+
+This will save quantized weights in the following folder structure:
+
+```
+output_dir/
+├── _layer_0/
+│   ├── _numa_0/
+│   │   ├── INT4_down_0_*.kt
+│   │   ├── INT4_gate_0_*.kt
+│   │   └── INT4_up_0_*.kt
+│   └── _numa_1/
+│       └── ...
+├── _layer_1/
+│   └── ...
+└── ...
+```
+
+**When to use `--no-merge-safetensor`:**
+- Machine runs out of memory during the merge step
+- Need to process very large models on memory-constrained systems
+- Want to preserve intermediate layer-wise quantized weights
+
+### Resume Layer
+
+For memory-constrained systems that are unable to complete quantization despite enabling low memory mode with `--no-merge-safetensor`, restart the script with the `--resume-layer` arg to specify the layer from which to continue the conversion process. In the example below, we skip layers 0-11 and resume conversion starting with layer 12.
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /path/to/model \
+  --input-type bf16 \
+  --output /path/to/output \
+  --quant-method int4 \
+  --no-merge-safetensor
+  --resume-layer 12
+```
+
+## Examples
+
+### Example 1: Quantize DeepSeek-V3.1 (FP8 → INT4)
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /mnt/data/models/DeepSeek-V3.1 \
+  --input-type fp8 \
+  --output /mnt/data/models/DeepSeek-V3.1-INT4 \
+  --quant-method int4 \
+  --cpuinfer-threads 60 \
+  --threadpool-count 2
+```
+
+### Example 2: Quantize Qwen3-Next-80B (BF16 → INT4, Low Memory)
+
+```bash
+python scripts/convert_cpu_weights.py \
+  --input-path /mnt/data/models/Qwen3-Next-80B-A3B-Instruct \
+  --input-type bf16 \
+  --output /mnt/data/models/Qwen3-Next-80B-A3B-Instruct-INT4 \
+  --quant-method int4 \
+  --cpuinfer-threads 60 \
+  --threadpool-count 2 \
+  --no-merge-safetensor
+```
+
+---
+
+## GPU Weight Quantization
+
+### Prerequisites
+
+GPU weight quantization requires additional dependencies. Install them before proceeding:
+
+```bash
+pip install accelerate transformers llmcompressor datasets
+```
+
+**Required packages:**
+- `accelerate`: For distributed model loading and device mapping
+- `transformers`: For model and tokenizer loading
+- `llmcompressor`: For quantization (supports GPTQ and RTN methods)
+- `datasets`: For calibration data loading (GPTQ only)
+
+**Documentation:** This tool is based on llmcompressor. For more details, see [llmcompressor quantization guide](https://docs.vllm.ai/projects/llm-compressor/en/latest/getting-started/compress/#select-a-quantization-method-and-scheme).
+
+### Overview
+
+Apply weight quantization to model weights for GPU-resident "hot" experts (frequently accessed) in CPU-GPU hybrid inference. This tool works together with `convert_cpu_weights.py` to enable heterogeneous expert placement:
+
+- **GPU-resident experts** ("hot" experts) use GPTQ/RTN quantization (this tool) for efficient GPU memory usage
+- **CPU-resident experts** ("cold" experts) use AMX-optimized INT4/INT8 quantization (convert_cpu_weights.py)
+- **Attention layers, gates, and shared experts** remain in higher precision
+
+This approach maximizes throughput and resource utilization by intelligently distributing experts across CPUs and GPUs.
+
+### Quantization Methods
+
+#### 1. GPTQ (Calibration-based, Default)
+**Pros:**
+- Higher accuracy through calibration-based quantization
+- Recommended for production deployments
+
+**Cons:**
+- Requires calibration dataset
+- Slower quantization process
+- Higher memory requirements (needs Hessian matrix)
+
+#### 2. RTN (Round-To-Nearest)
+**Pros:**
+- Fast quantization (no calibration needed)
+- Lower memory requirements
+- Good for quick testing and prototyping
+
+**Cons:**
+- Slightly lower accuracy compared to GPTQ
+- No calibration optimization
+
+### Quantization Types
+
+- **W4A16**: 4-bit weights, 16-bit activations (INT4)
+- **W8A16**: 8-bit weights, 16-bit activations (INT8)
+
+### Basic Usage
+
+#### GPTQ Quantization (Recommended for Production)
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /path/to/model \
+  --output_dir /path/to/output \
+  --quant_method GPTQ \
+  --quant_type W4A16
+```
+
+#### RTN Quantization (Fast, for Testing)
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /path/to/model \
+  --output_dir /path/to/output \
+  --quant_method RTN \
+  --quant_type W4A16
+```
+
+### Memory Requirements
+
+Understanding memory requirements is crucial for successful quantization. The requirements differ significantly between RTN and GPTQ methods.
+
+#### RTN Memory Requirements
+
+RTN only requires memory for quantization parameters (scales/zero-points):
+
+| Component | Requirement |
+|-----------|-------------|
+| **DRAM (CPU Memory)** | ≥ Total model parameters |
+| **VRAM (GPU Memory)** | ≥ Single layer parameters |
+
+**Example: DeepSeek-R1-0528-BF16 (684B parameters)**
+- DRAM: ~1368 GB (684B params × 2 bytes)
+- VRAM: ~22.4 GB (1 layer)
+
+#### GPTQ Memory Requirements
+
+GPTQ requires additional memory for Hessian matrices during calibration:
+
+| Component | Requirement |
+|-----------|-------------|
+| **DRAM (CPU Memory)** | ≥ Total model parameters |
+| **VRAM (GPU Memory)** | ≥ Single layer parameters × 2 |
+
+The Hessian matrix is approximately the same size as the layer weights and is used to increase accuracy recovery.
+
+**Example: DeepSeek-R1-0528-BF16 (684B parameters)**
+- DRAM: ~1368 GB (684B params × 2 bytes)
+- VRAM: ~44.8 GB (1 layer × 2 for Hessian matrix)
+
+#### Method Comparison
+
+| Method | Speed | VRAM | Accuracy | Use Case |
+|--------|-------|------|----------|----------|
+| **RTN** | Fast | Low (~22GB) | Good | Testing, prototyping |
+| **GPTQ** | Slow | High (~45GB) | Better | Production deployment |
+
+### Advanced Options
+
+#### Calibration Configuration (GPTQ Only)
+
+For GPTQ quantization, control the calibration process for better quantization quality:
+
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /path/to/model \
+  --output_dir /path/to/output \
+  --quant_method GPTQ \
+  --quant_type W4A16 \
+  --num_calibration_samples 512 \
+  --max_sequence_length 2048 \
+  --dataset HuggingFaceH4/ultrachat_200k \
+  --dataset_split train_sft
+```
+
+**Options (GPTQ only):**
+- `--num_calibration_samples`: Number of samples for calibration (default: 512)
+- `--max_sequence_length`: Maximum sequence length (default: 2048)
+- `--dataset`: HuggingFace dataset for calibration
+- `--dataset_split`: Dataset split to use
+- `--dampening_frac`: Dampening fraction to reduce quantization noise (default: 0.1)
+
+#### Memory Management
+
+Use `--max_gpu_memory` to limit GPU memory usage and offload remaining layers to CPU:
+
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /path/to/model \
+  --output_dir /path/to/output \
+  --quant_method GPTQ \
+  --quant_type W4A16 \
+  --max_gpu_memory "40GiB"
+```
+
+**Recommended settings for GPTQ:**
+
+| GPU VRAM | Suggested `--max_gpu_memory` | Notes |
+|----------|------------------------------|-------|
+| 24 GiB   | 10-12 GiB | Reserve ~50% for Hessian |
+| 48 GiB   | 24-30 GiB | Reserve ~40% for Hessian |
+| 80 GiB   | 40-50 GiB | Reserve ~40% for Hessian |
+
+**Recommended settings for RTN:**
+
+| GPU VRAM | Suggested `--max_gpu_memory` | Notes |
+|----------|------------------------------|-------|
+| 24 GiB   | 18-20 GiB | No Hessian needed |
+| 48 GiB   | 40-45 GiB | No Hessian needed |
+| 80 GiB   | 70-75 GiB | No Hessian needed |
+
+**Options:**
+- `--max_gpu_memory`: Maximum GPU memory for model weights per device (e.g., '40GiB')
+- `--max_cpu_memory`: Maximum CPU memory (default: 1000GiB when `--max_gpu_memory` is set)
+
+**Important:** llmcompressor does not support disk offloading. Ensure your machine has enough GPU + CPU memory to load the entire model. If you still encounter OOM:
+1. Use RTN instead of GPTQ (requires less memory)
+2. Reduce `--num_calibration_samples` (GPTQ only, e.g., 256)
+3. Reduce `--max_sequence_length` (GPTQ only, e.g., 1024)
+4. Use `--force_cpu` to run entirely on CPU (slower but avoids GPU OOM)
+
+### Examples
+
+#### Example 1: GPTQ Quantization for Production (Qwen3-Next-80B, W4A16)
+
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /mnt/data/models/Qwen3-Next-80B-A3B-Instruct \
+  --output_dir /mnt/data/models/Qwen3-Next-80B-A3B-Instruct-GPTQ-W4A16 \
+  --quant_method GPTQ \
+  --quant_type W4A16 \
+  --num_calibration_samples 512 \
+  --max_sequence_length 2048 \
+  --max_gpu_memory "40GiB" \
+  --trust_remote_code
+```
+
+#### Example 2: RTN Quantization for Fast Testing (DeepSeek-R1, W4A16)
+
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /mnt/data/models/DeepSeek-R1-0528-BF16 \
+  --output_dir /mnt/data/models/DeepSeek-R1-0528-RTN-W4A16 \
+  --quant_method RTN \
+  --quant_type W4A16 \
+  --max_gpu_memory "70GiB" \
+  --trust_remote_code
+```
+
+#### Example 3: GPTQ with Custom Calibration Dataset (GLM-4.5-Air, W8A16)
+
+```bash
+python scripts/convert_gpu_weights.py \
+  --model_id /mnt/data/models/GLM-4.5-Air \
+  --output_dir /mnt/data/models/GLM-4.5-Air-GPTQ-W8A16 \
+  --quant_method GPTQ \
+  --quant_type W8A16 \
+  --dataset "tatsu-lab/alpaca" \
+  --dataset_split "train" \
+  --num_calibration_samples 256 \
+  --max_gpu_memory "40GiB" \
+  --trust_remote_code
+```
@@ -0,0 +1,277 @@
+import os
+
+# insert the path of the project
+import sys
+
+# sys.path.insert(0, "/home/azure/ktransformers")
+import argparse
+import torch
+from safetensors import safe_open
+from safetensors.torch import save_file
+import re
+from collections import defaultdict
+import itertools
+import os
+import torch
+import numpy as np
+
+tensor_from_amx = [".mlp.experts."]  # todo: add keys in gguf that should be used in the final tensor
+
+
+def safe_open_binary_to_tensor(file_path):
+    if not os.path.exists(file_path):
+        raise FileNotFoundError(f"文件不存在: {file_path}")
+
+    if not os.access(file_path, os.R_OK):
+        raise PermissionError(f"没有权限读取文件: {file_path}")
+
+    try:
+        with open(file_path, "rb") as f:
+            binary_data = f.read()
+
+        np_array = np.frombuffer(binary_data, dtype=np.int8)
+
+        tensor = torch.from_numpy(np_array)
+
+        return tensor
+
+    except Exception as e:
+        raise IOError(f"file process error: {str(e)}")
+
+
+def read_safetensor_keys_from_folder(folder_path) -> dict:
+    """
+    :param folder_path: folder path
+    :return: key_to_file_map
+    """
+    # check if the folder path is exist
+    if not os.path.exists(folder_path):
+        raise FileNotFoundError(f"GGUF dir not found: {folder_path}")
+    if os.path.isfile(folder_path):
+        folder_path = os.path.dirname(folder_path)
+
+    key_to_file_map = {}
+
+    found_safetensor = False
+    for root, dirs, files in os.walk(folder_path):
+        # sort files
+        files = sorted(files)
+        for file in files:
+            if file.endswith(".safetensors"):
+                found_safetensor = True
+                file_path = os.path.join(root, file)
+                try:
+                    with safe_open(file_path, framework="pt") as f:
+                        for key in f.keys():
+                            if "model.layers.61" in key:
+                                # skip MTP layer
+                                continue
+                            # try:
+                            #     if int(key.split('.')[2]) > 4:
+                            #         continue
+                            # except:
+                            #     pass
+                            key_to_file_map[key] = file_path
+                except Exception as e:
+                    print(f"Error reading Safetensor file {file_path}: {e}")
+
+    if not found_safetensor:
+        raise FileNotFoundError(f"No Safetensor files found in {folder_path}")
+
+    return key_to_file_map
+
+
+def read_amx_tensor_from_folder(folder_path, keys) -> dict:
+    layer_list = [f"_layer_{i}" for i in range(3, 61)]
+    numa_list = ["_numa_0", "_numa_1"]
+
+    down_list = [f"INT4_down_{i}_quant_.kt" for i in range(256)]
+    gate_list = [f"INT4_gate_{i}_quant_.kt" for i in range(256)]
+    up_list = [f"INT4_up_{i}_quant_.kt" for i in range(256)]
+    down_scale_list = [f"INT4_down_{i}_scale_.kt" for i in range(256)]
+    gate_scale_list = [f"INT4_gate_{i}_scale_.kt" for i in range(256)]
+    up_scale_list = [f"INT4_up_{i}_scale_.kt" for i in range(256)]
+    target = ["ffn_up_exps", "ffn_down_exps", "ffn_gate_exps"]
+    tensor_file_map = {}
+    for key in keys:
+        layer = int(key.split(".")[1])
+        if layer < 3:
+            continue
+        layer_path = f"_layer_{layer}"
+        # concatenate the path layer/numa/(down|gate|up)_(0-255)_3670016Byte_quant_.kt
+        # store the path in the tensor_file_map
+        # key = key+'.idx.weight'
+        # scale_key = key+'.idx.scale'
+        for numa_idx, numa in enumerate(numa_list):
+            # TODO: 256 should be a variable
+            for i in range(256):
+                prefix_key = ".".join(key.split(".")[:-1])
+
+                experts_key = prefix_key + f".{i}.numa.{numa_idx}.weight"
+                scale_key = prefix_key + f".{i}.numa.{numa_idx}.scale"
+                if "down" in experts_key:
+                    tensor_file_map[experts_key] = os.path.join(folder_path, layer_path, numa, down_list[i])
+                    tensor_file_map[scale_key] = os.path.join(folder_path, layer_path, numa, down_scale_list[i])
+                elif "gate" in experts_key:
+                    tensor_file_map[experts_key] = os.path.join(folder_path, layer_path, numa, gate_list[i])
+                    tensor_file_map[scale_key] = os.path.join(folder_path, layer_path, numa, gate_scale_list[i])
+                elif "up" in experts_key:
+                    tensor_file_map[experts_key] = os.path.join(folder_path, layer_path, numa, up_list[i])
+                    tensor_file_map[scale_key] = os.path.join(folder_path, layer_path, numa, up_scale_list[i])
+    return tensor_file_map
+
+
+# def translate_name(name:str)->str:
+#     """
+#     :param name: name of the tensor
+#     :return: translated name
+#     """
+#     name = translate_name_to_gguf(name)
+#     name = name.replace(".up_proj.", ".ffn_up_exps.")
+#     name = name.replace(".down_proj.", ".ffn_down_exps.")
+#     name = name.replace(".gate_proj.", ".ffn_gate_exps.")
+#     name = name.replace(".ffn_gate_inp.e_score_correction_bias", ".exp_probs_b.bias")
+#     return name
+
+
+def _clean_keys(keys):
+    keys = list(keys)
+    target = ["ffn_up_exps", "ffn_down_exps", "ffn_gate_exps"]
+    # only keep the keys that contain the target
+    keys = [key for key in keys if any(target_key in key for target_key in target) and "ggml_type" not in key]
+    return keys
+
+
+def combine_tensor_sources(safetensor_path, amx_path):
+    safetensor_tensor_file_map = read_safetensor_keys_from_folder(safetensor_path)
+
+    keys = _clean_keys(safetensor_tensor_file_map.keys())
+
+    amx_tensor_file_map = read_amx_tensor_from_folder(amx_path, keys)
+    target_tensor_map = {}
+    for key in safetensor_tensor_file_map.keys():
+        if "_exps." in key:
+            continue
+
+        target_tensor_map[key] = safetensor_tensor_file_map[key]
+
+    for key in amx_tensor_file_map.keys():
+        target_tensor_map[key] = amx_tensor_file_map[key]
+
+    return target_tensor_map
+
+
+def write_combined_tensor(target_tensor_map: dict, output_path: str):
+    # Ensure output directory exists
+    os.makedirs(output_path, exist_ok=True)
+
+    # Cache for safetensor file handles and GGUF loaders
+    safetensors_cache = {}
+    amx_cache = {}
+
+    # Group tensors by layer
+    layer_groups = defaultdict(list)
+    non_layer_keys = []
+    layer_pattern = re.compile(r"blk\.(\d+)\.")
+
+    for key in target_tensor_map:
+        match = layer_pattern.search(key)
+        if match:
+            layer_groups[int(match.group(1))].append(key)
+        else:
+            non_layer_keys.append(key)
+
+    # Calculate the number of shards
+    total_shards = len(layer_groups) + (1 if non_layer_keys else 0) - 1
+
+    shard_idx = 0
+    # Save non-layer tensors to the first shard if they exist
+    if non_layer_keys:
+        tensors = {}
+        for key in non_layer_keys:
+            file_path = target_tensor_map[key]
+            tensor = None
+            ggml_type = None
+            if file_path.endswith(".safetensors"):
+                if file_path not in safetensors_cache:
+                    safetensors_cache[file_path] = safe_open(file_path, framework="pt")
+                f = safetensors_cache[file_path]
+                tensor = f.get_tensor(key)
+            elif file_path.endswith(".kt"):
+                tensor = safe_open_binary_to_tensor(file_path)
+            else:
+                raise ValueError(f"Unsupported file format: {file_path}")
+            tensors[key] = tensor
+
+        output_file = os.path.join(output_path, f"model-{shard_idx:05}-of-{total_shards:05}.safetensors")
+        print(f"Saving non-layer tensors to {output_file}")
+        save_file(tensors, output_file)
+        shard_idx += 1
+
+    # Save each layer's tensors to subsequent shards
+    for layer_num in sorted(layer_groups.keys()):
+        layer_keys = layer_groups[layer_num]
+        tensors = {}
+        for key in layer_keys:
+            file_path = target_tensor_map[key]
+            tensor = None
+            ggml_type = None
+            if file_path.endswith(".safetensors"):
+                if file_path not in safetensors_cache:
+                    safetensors_cache[file_path] = safe_open(file_path, framework="pt")
+                f = safetensors_cache[file_path]
+                tensor = f.get_tensor(key)
+                tensor_info = tensor.shape
+            elif file_path.endswith(".kt"):
+                tensor = safe_open_binary_to_tensor(file_path)
+            else:
+                raise ValueError(f"Unsupported file format: {file_path}")
+            tensors[key] = tensor
+
+        output_file = os.path.join(output_path, f"model-{shard_idx:05}-of-{total_shards:05}.safetensors")
+        print(f"Saving layer {layer_num} to {output_file}")
+        save_file(tensors, output_file)
+        shard_idx += 1
+    return
+
+
+def main():
+    # 输入已经处理过的混合模型路径，提前处理好的amx路径，输出路径
+    parser = argparse.ArgumentParser(description="Read parameters from Safetensor and GGUF files")
+    parser.add_argument(
+        "--safetensor_path",
+        type=str,
+        help="Path to the Safetensor file",
+        default="/mnt/data/models/DeepSeek-R1-GGML-FP8-Hybrid/DeepSeek-R1-IQ1S-FP8",
+    )
+    parser.add_argument(
+        "--amx_path", type=str, help="Path to the GGUF file", default="/mnt/data/models/DeepSeek-R1-INT4"
+    )
+    parser.add_argument(
+        "--output_path",
+        type=str,
+        help="Path to the output file",
+        default="/mnt/data/models/DeepSeek-R1-GGML-FP8-Hybrid/DeepSeek-R1-AMXQ4-FP8",
+    )
+
+    # print all the arguments
+    print("All the arguments:")
+    print(parser.parse_args())
+
+    # 解析命令行参数
+    args = parser.parse_args()
+
+    safetensor_path = args.safetensor_path
+    amx_path = args.amx_path
+    output_path = args.output_path
+
+    target_tensor_map = combine_tensor_sources(safetensor_path, amx_path)
+    for key, value in target_tensor_map.items():
+        print(f"{key}: {value}")
+    write_combined_tensor(target_tensor_map, output_path)
+
+    return
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+"""
+CPU feature detection script for kt-kernel.
+
+This script checks if your CPU supports the required instruction sets for FP8 MoE:
+- AVX512F (foundation)
+- AVX512_BF16 (BF16 dot product)
+- AVX512_VNNI (VNNI instructions)
+- AVX512_VBMI (byte permutation)
+
+Usage:
+    python3 scripts/check_cpu_features.py
+"""
+
+import os
+import sys
+
+
+def check_cpuinfo():
+    """Check CPU features via /proc/cpuinfo."""
+    try:
+        with open("/proc/cpuinfo", "r") as f:
+            cpuinfo = f.read().lower()
+        return cpuinfo
+    except FileNotFoundError:
+        return None
+
+
+def main():
+    print("=" * 70)
+    print("KT-Kernel CPU Feature Detection")
+    print("=" * 70)
+    print()
+
+    cpuinfo = check_cpuinfo()
+
+    if cpuinfo is None:
+        print("❌ /proc/cpuinfo not found (not on Linux?)")
+        print("   Cannot detect CPU features automatically.")
+        sys.exit(1)
+
+    # Extract CPU model
+    for line in cpuinfo.split("\n"):
+        if "model name" in line:
+            model = line.split(":")[1].strip()
+            print(f"CPU Model: {model}")
+            break
+    print()
+
+    # Check AMX support
+    print("AMX Support (Intel Sapphire Rapids+):")
+    amx_flags = ["amx_tile", "amx_int8", "amx_bf16"]
+    amx_status = {}
+    for flag in amx_flags:
+        has_flag = flag in cpuinfo
+        amx_status[flag] = has_flag
+        status = "✅" if has_flag else "❌"
+        print(f"  {status} {flag.upper()}")
+
+    has_amx = all(amx_status.values())
+    print(f"\n  Overall AMX Support: {'✅ YES' if has_amx else '❌ NO'}")
+    print()
+
+    # Check AVX512 support
+    print("AVX512 Support (required for FP8 MoE):")
+    avx512_flags = ["avx512f", "avx512_bf16", "avx512_vnni", "avx512_vbmi"]
+    avx512_status = {}
+    for flag in avx512_flags:
+        has_flag = flag in cpuinfo
+        avx512_status[flag] = has_flag
+        status = "✅" if has_flag else "❌"
+        flag_desc = {
+            "avx512f": "AVX512F (foundation)",
+            "avx512_bf16": "AVX512_BF16 (BF16 dot product)",
+            "avx512_vnni": "AVX512_VNNI (VNNI instructions)",
+            "avx512_vbmi": "AVX512_VBMI (byte permutation)",
+        }
+        print(f"  {status} {flag_desc.get(flag, flag.upper())}")
+
+    has_avx512_full = all(avx512_status.values())
+    print(f"\n  Overall AVX512 Support: {'✅ YES' if has_avx512_full else '❌ NO'}")
+
+    if not has_avx512_full and avx512_status["avx512f"]:
+        missing = [f for f in avx512_flags if not avx512_status[f]]
+        print(f"  ⚠️  Warning: AVX512F detected but missing: {', '.join(missing)}")
+        print(f"      kt-kernel will fall back to AVX2 mode")
+    print()
+
+    # Check AVX2 support
+    print("AVX2 Support (fallback):")
+    has_avx2 = "avx2" in cpuinfo
+    status = "✅" if has_avx2 else "❌"
+    print(f"  {status} AVX2")
+    print()
+
+    # Recommendation
+    print("=" * 70)
+    print("Recommendation:")
+    print("=" * 70)
+    if has_amx:
+        print("✅ Your CPU supports AMX - you can use the highest performance mode!")
+        print("   Build with: -DKTRANSFORMERS_CPU_USE_AMX_AVX512=ON -DKTRANSFORMERS_CPU_USE_AMX=ON")
+    elif has_avx512_full:
+        print("✅ Your CPU supports full AVX512 (F/BF16/VNNI/VBMI) - FP8 MoE will work!")
+        print("   Build with: -DKTRANSFORMERS_CPU_USE_AMX_AVX512=ON")
+    elif avx512_status.get("avx512f", False):
+        print("⚠️  Your CPU has AVX512F but missing required extensions.")
+        print("   FP8 MoE will NOT work. kt-kernel will fall back to AVX2 mode.")
+        print("   Missing extensions:", ", ".join([f for f in avx512_flags if not avx512_status.get(f, False)]))
+    elif has_avx2:
+        print("ℹ️  Your CPU supports AVX2 only - basic compatibility mode.")
+        print("   FP8 MoE will NOT be available, but other features will work.")
+    else:
+        print("❌ Your CPU does not support the minimum required instruction set (AVX2).")
+        print("   kt-kernel may not work on this system.")
+    print()
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,529 @@
+#!/usr/bin/env python3
+"""
+Compare two sets of quantized weights generated by convert_cpu_weights.py
+
+This script supports comparing:
+- Two safetensor format weights (merged)
+- Two .kt format weights (layer folder structure)
+- One safetensor and one .kt format (cross-format comparison)
+
+Usage:
+    python compare_weights.py --path1 /path/to/weights1 --path2 /path/to/weights2
+    python compare_weights.py --path1 /path/to/weights1 --path2 /path/to/weights2 --tolerance 1e-5
+"""
+
+import argparse
+import os
+import glob
+import numpy as np
+import torch
+from safetensors import safe_open
+from typing import Dict, Tuple
+from collections import defaultdict
+
+
+def unpack_awq_int32_to_int8(packed: np.ndarray, bits: int = 4) -> np.ndarray:
+    """Unpack AWQ int32 packed format to int8
+
+    AWQ uses INT4 quantization: 8 x 4-bit values packed into 1 x 32-bit integer
+
+    Args:
+        packed: Packed int32 array
+        bits: Number of bits per element (default: 4)
+
+    Returns:
+        Unpacked int8 array
+    """
+    if packed.dtype != np.int32:
+        # Try to reinterpret as int32
+        packed = packed.view(np.int32)
+
+    pack_num = 32 // bits  # 8 for INT4
+    unpacked_size = packed.size * pack_num
+
+    unpacked = np.empty(unpacked_size, dtype=np.int8)
+
+    for i in range(pack_num):
+        shift = i * bits
+        mask = (1 << bits) - 1  # 0x0F for 4-bit
+        unpacked[i::pack_num] = ((packed >> shift) & mask).astype(np.int8)
+
+    return unpacked
+
+
+def normalize_tensor_dtype(tensor: np.ndarray, tensor_name: str, is_awq: bool = False) -> np.ndarray:
+    """Normalize tensor to consistent dtype based on tensor type
+
+    Args:
+        tensor: Input tensor
+        tensor_name: Name of the tensor (used to determine type)
+        is_awq: Whether this is AWQ format (requires unpacking)
+
+    Returns:
+        Normalized tensor with consistent dtype
+    """
+    # Determine tensor type from name
+    is_scale = "scale" in tensor_name
+    is_weight = "weight" in tensor_name
+    is_qzeros = "qzeros" in tensor_name
+
+    if is_scale:
+        # Scale should be float32
+        if tensor.dtype != np.float32:
+            # Try to reinterpret bytes as float32
+            tensor = tensor.view(np.float32)
+        return tensor
+
+    elif is_weight or is_qzeros:
+        # Weight/qzeros should be int8
+        if is_awq and tensor.dtype == np.int32:
+            # AWQ format: unpack int32 to int8
+            tensor = unpack_awq_int32_to_int8(tensor)
+        elif tensor.dtype == np.float32:
+            # Two cases for float32:
+            # Case 1: Values look like int8 values (e.g., [37., 73., -70.])
+            #         -> use astype to convert values
+            # Case 2: Values are large scientific notation (e.g., [2.6e34, ...])
+            #         -> use view to reinterpret bytes
+
+            # Check if values are in int8 range (-128 to 127)
+            if len(tensor) > 0:
+                sample_size = min(100, len(tensor))
+                sample_values = tensor.flat[:sample_size]
+
+                # If most values are in int8 range and have no decimal parts
+                in_int8_range = np.all((sample_values >= -128) & (sample_values <= 127))
+                is_integer_valued = np.all(sample_values == np.round(sample_values))
+
+                if in_int8_range and is_integer_valued:
+                    # Case 1: Direct value conversion
+                    tensor = tensor.astype(np.int8)
+                else:
+                    # Case 2: Byte reinterpretation (4 bytes -> 4 int8s)
+                    tensor = tensor.view(np.int8)
+            else:
+                tensor = tensor.astype(np.int8)
+
+        elif tensor.dtype == np.int32:
+            # Reinterpret int32 as int8 (4x more elements)
+            tensor = tensor.view(np.int8)
+        elif tensor.dtype != np.int8:
+            # Other types: try to convert
+            tensor = tensor.astype(np.int8)
+
+        return tensor
+
+    else:
+        # Unknown type, return as-is
+        return tensor
+
+
+def load_kt_binary(file_path: str) -> np.ndarray:
+    """Load .kt format binary tensor file
+
+    Args:
+        file_path: Path to .kt binary file
+
+    Returns:
+        numpy array with the loaded tensor
+    """
+    if not os.path.exists(file_path):
+        raise FileNotFoundError(f"File not found: {file_path}")
+
+    with open(file_path, "rb") as f:
+        binary_data = f.read()
+
+    # Determine dtype based on file name
+    if "scale" in file_path:
+        dtype = np.float32
+    else:
+        dtype = np.int8
+
+    return np.frombuffer(binary_data, dtype=dtype)
+
+
+def detect_weight_format(path: str) -> str:
+    """Detect if weights are in safetensor or .kt format
+
+    Args:
+        path: Path to weight directory
+
+    Returns:
+        'safetensor' or 'kt' or 'unknown'
+    """
+    if not os.path.exists(path):
+        raise FileNotFoundError(f"Path not found: {path}")
+
+    # Check for safetensor files
+    safetensor_files = glob.glob(os.path.join(path, "*.safetensors"))
+    if safetensor_files:
+        return "safetensor"
+
+    # Check for layer folder structure
+    layer_folders = glob.glob(os.path.join(path, "_layer_*"))
+    if layer_folders:
+        return "kt"
+
+    return "unknown"
+
+
+def detect_awq_format(weights_sample: Dict[str, np.ndarray]) -> bool:
+    """Detect if weights are in AWQ format
+
+    AWQ format characteristics:
+    - Has 'qzeros' tensors
+    - Weight tensors are int32 dtype (packed)
+
+    Args:
+        weights_sample: Sample of loaded weights
+
+    Returns:
+        True if AWQ format detected
+    """
+    has_qzeros = any("qzeros" in key for key in weights_sample.keys())
+
+    if not has_qzeros:
+        return False
+
+    # Check if weight tensors are int32
+    for key, tensor in weights_sample.items():
+        if "weight" in key and tensor.dtype == np.int32:
+            return True
+
+    return False
+
+
+def load_safetensor_weights(path: str) -> Dict[str, np.ndarray]:
+    """Load all weights from safetensor format
+
+    Args:
+        path: Path to directory containing safetensor files
+
+    Returns:
+        Dictionary mapping tensor names to numpy arrays (dtype normalized)
+    """
+    weights = {}
+
+    safetensor_files = sorted(glob.glob(os.path.join(path, "*.safetensors")))
+    if not safetensor_files:
+        raise FileNotFoundError(f"No safetensor files found in {path}")
+
+    print(f"Loading safetensor files from {path}")
+
+    # First pass: load all tensors
+    for file in safetensor_files:
+        with safe_open(file, framework="pt") as f:
+            for key in f.keys():
+                # Only load MoE expert weights for comparison
+                if ".ffn_" in key and "_exps." in key:
+                    tensor = f.get_tensor(key)
+                    weights[key] = tensor.cpu().numpy()
+
+    # Detect AWQ format
+    is_awq = detect_awq_format(weights)
+    print(f"  Format detected: {'AWQ' if is_awq else 'INT4/INT8'}")
+
+    # Second pass: normalize dtypes
+    print(f"  Normalizing dtypes...")
+    for key in list(weights.keys()):
+        original_dtype = weights[key].dtype
+        original_shape = weights[key].shape
+        weights[key] = normalize_tensor_dtype(weights[key], key, is_awq=is_awq)
+
+        if weights[key].shape != original_shape or weights[key].dtype != original_dtype:
+            print(f"    {key}: {original_dtype}{original_shape} -> {weights[key].dtype}{weights[key].shape}")
+
+    print(f"  Loaded {len(weights)} tensors from safetensor format")
+    return weights
+
+
+def load_kt_weights(path: str) -> Dict[str, np.ndarray]:
+    """Load all weights from .kt format (layer folder structure)
+
+    Args:
+        path: Path to directory containing _layer_* folders
+
+    Returns:
+        Dictionary mapping tensor names to numpy arrays
+    """
+    weights = {}
+
+    layer_folders = sorted(glob.glob(os.path.join(path, "_layer_*")))
+    if not layer_folders:
+        raise FileNotFoundError(f"No _layer_* folders found in {path}")
+
+    print(f"Loading .kt files from {path}")
+
+    for layer_folder in layer_folders:
+        # Extract layer index from folder name
+        layer_idx = int(os.path.basename(layer_folder).split("_")[-1])
+
+        # Find all NUMA folders
+        numa_folders = sorted(glob.glob(os.path.join(layer_folder, "_numa_*")))
+
+        for numa_folder in numa_folders:
+            # Extract NUMA index
+            numa_idx = int(os.path.basename(numa_folder).split("_")[-1])
+
+            # Find all .kt files
+            kt_files = glob.glob(os.path.join(numa_folder, "*.kt"))
+
+            for kt_file in kt_files:
+                filename = os.path.basename(kt_file)
+
+                # Parse filename to extract metadata
+                # Format: {METHOD}_{proj}_{expert}_{size}Byte_{type}_.kt
+                parts = filename.replace(".kt", "").split("_")
+
+                if len(parts) >= 5:
+                    method = parts[0]  # INT4, INT8, etc.
+                    proj = parts[1]  # down, gate, up
+                    expert = parts[2]  # expert ID
+                    tensor_type = parts[4]  # quant or scale
+
+                    # Map proj names
+                    proj_map = {"down": "ffn_down_exps", "gate": "ffn_gate_exps", "up": "ffn_up_exps"}
+
+                    proj_key = proj_map.get(proj, proj)
+
+                    # Build key matching safetensor format
+                    if tensor_type == "quant":
+                        key = f"blk.{layer_idx}.{proj_key}.{expert}.numa.{numa_idx}.weight"
+                    else:  # scale
+                        key = f"blk.{layer_idx}.{proj_key}.{expert}.numa.{numa_idx}.scale"
+
+                    # Load tensor
+                    weights[key] = load_kt_binary(kt_file)
+
+    # Normalize dtypes (.kt format is never AWQ)
+    print(f"  Normalizing dtypes...")
+    for key in list(weights.keys()):
+        original_dtype = weights[key].dtype
+        original_shape = weights[key].shape
+        weights[key] = normalize_tensor_dtype(weights[key], key, is_awq=False)
+
+        if weights[key].shape != original_shape or weights[key].dtype != original_dtype:
+            print(f"    {key}: {original_dtype}{original_shape} -> {weights[key].dtype}{weights[key].shape}")
+
+    print(f"  Loaded {len(weights)} tensors from .kt format")
+    return weights
+
+
+def normalize_key(key: str) -> Tuple[int, str, int, str]:
+    """Normalize tensor key to extract layer, projection, expert, and type
+
+    Args:
+        key: Tensor key like "blk.0.ffn_up_exps.5.weight" or "blk.0.ffn_up_exps.5.numa.0.weight"
+
+    Returns:
+        Tuple of (layer_idx, proj_name, expert_idx, tensor_type)
+    """
+    parts = key.split(".")
+
+    layer_idx = int(parts[1])
+    proj_name = parts[2]
+    expert_idx = int(parts[3])
+
+    # Handle both formats: with and without numa
+    if "numa" in key:
+        tensor_type = parts[6]  # weight or scale
+    else:
+        tensor_type = parts[4]  # weight, scale, or qzeros
+
+    return (layer_idx, proj_name, expert_idx, tensor_type)
+
+
+def compare_weights(
+    weights1: Dict[str, np.ndarray], weights2: Dict[str, np.ndarray], tolerance: float = 1e-6
+) -> Tuple[bool, Dict[str, Dict]]:
+    """Compare two sets of weights
+
+    Args:
+        weights1: First set of weights
+        weights2: Second set of weights
+        tolerance: Numerical tolerance for comparison
+
+    Returns:
+        Tuple of (all_match, differences_dict)
+    """
+    print("\n" + "=" * 80)
+    print("WEIGHT COMPARISON")
+    print("=" * 80)
+
+    # Group keys by normalized form (ignoring numa index)
+    def group_by_base_key(weights):
+        groups = defaultdict(list)
+        for key in weights.keys():
+            try:
+                layer, proj, expert, ttype = normalize_key(key)
+                base_key = f"blk.{layer}.{proj}.{expert}.{ttype}"
+                groups[base_key].append(key)
+            except:
+                # Skip keys that don't match expected format
+                pass
+        return groups
+
+    groups1 = group_by_base_key(weights1)
+    groups2 = group_by_base_key(weights2)
+
+    all_base_keys = sorted(set(groups1.keys()) | set(groups2.keys()))
+
+    all_match = True
+    differences = {}
+
+    total_comparisons = 0
+    matching_comparisons = 0
+
+    for base_key in all_base_keys:
+        keys1 = groups1.get(base_key, [])
+        keys2 = groups2.get(base_key, [])
+
+        if not keys1:
+            print(f"❌ Missing in weights1: {base_key}")
+            differences[base_key] = {"status": "missing_in_weights1"}
+            all_match = False
+            continue
+
+        if not keys2:
+            print(f"❌ Missing in weights2: {base_key}")
+            differences[base_key] = {"status": "missing_in_weights2"}
+            all_match = False
+            continue
+
+        # For kt format, we may have multiple keys (one per NUMA node)
+        # We need to concatenate them for comparison
+        if len(keys1) > 1 or len(keys2) > 1:
+            # Concatenate tensors from all NUMA nodes
+            tensor1 = np.concatenate([weights1[k] for k in sorted(keys1)])
+            tensor2 = np.concatenate([weights2[k] for k in sorted(keys2)])
+        else:
+            tensor1 = weights1[keys1[0]]
+            tensor2 = weights2[keys2[0]]
+
+        total_comparisons += 1
+
+        # Debug: print dtype and shape info
+        if tensor1.dtype != tensor2.dtype:
+            print(f"⚠️  Dtype mismatch for {base_key}: {tensor1.dtype} vs {tensor2.dtype}")
+            print(f"   This should have been normalized. Shape: {tensor1.shape} vs {tensor2.shape}")
+
+        # Compare shapes
+        if tensor1.shape != tensor2.shape:
+            print(f"❌ Shape mismatch for {base_key}:")
+            print(f"   Shape1: {tensor1.shape} (dtype: {tensor1.dtype})")
+            print(f"   Shape2: {tensor2.shape} (dtype: {tensor2.dtype})")
+            differences[base_key] = {
+                "status": "shape_mismatch",
+                "shape1": tensor1.shape,
+                "shape2": tensor2.shape,
+                "dtype1": str(tensor1.dtype),
+                "dtype2": str(tensor2.dtype),
+            }
+            all_match = False
+            continue
+
+        # Compare dtypes (should be consistent after normalization)
+        if tensor1.dtype != tensor2.dtype:
+            print(f"❌ Dtype mismatch for {base_key} after normalization:")
+            print(f"   Dtype1: {tensor1.dtype}")
+            print(f"   Dtype2: {tensor2.dtype}")
+            differences[base_key] = {
+                "status": "dtype_mismatch",
+                "dtype1": str(tensor1.dtype),
+                "dtype2": str(tensor2.dtype),
+            }
+            all_match = False
+            continue
+
+        # Compare values
+        if np.allclose(tensor1, tensor2, atol=tolerance, rtol=tolerance):
+            matching_comparisons += 1
+        else:
+            max_diff = np.max(np.abs(tensor1 - tensor2))
+            mean_diff = np.mean(np.abs(tensor1 - tensor2))
+
+            print(f"❌ Value mismatch for {base_key}:")
+            print(f"   Max difference: {max_diff:.2e}")
+            print(f"   Mean difference: {mean_diff:.2e}")
+            print(f"   Tolerance: {tolerance:.2e}")
+
+            differences[base_key] = {
+                "status": "value_mismatch",
+                "max_diff": float(max_diff),
+                "mean_diff": float(mean_diff),
+                "tolerance": tolerance,
+            }
+            all_match = False
+
+    print("\n" + "=" * 80)
+    print("SUMMARY")
+    print("=" * 80)
+    print(f"Total comparisons: {total_comparisons}")
+    print(f"Matching: {matching_comparisons}")
+    print(f"Mismatching: {total_comparisons - matching_comparisons}")
+    print(f"Missing tensors: {len(differences) - (total_comparisons - matching_comparisons)}")
+
+    if all_match:
+        print("\n✅ All weights match!")
+    else:
+        print(f"\n❌ Found {len(differences)} differences")
+
+    return all_match, differences
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Compare two sets of quantized weights")
+    parser.add_argument("--path1", type=str, required=True, help="Path to first weight directory")
+    parser.add_argument("--path2", type=str, required=True, help="Path to second weight directory")
+    parser.add_argument(
+        "--tolerance", type=float, default=1e-6, help="Numerical tolerance for comparison (default: 1e-6)"
+    )
+
+    args = parser.parse_args()
+
+    # Validate paths
+    if not os.path.exists(args.path1):
+        print(f"Error: Path1 does not exist: {args.path1}")
+        return 1
+
+    if not os.path.exists(args.path2):
+        print(f"Error: Path2 does not exist: {args.path2}")
+        return 1
+
+    # Detect formats
+    print("Detecting weight formats...")
+    format1 = detect_weight_format(args.path1)
+    format2 = detect_weight_format(args.path2)
+
+    print(f"Path1 format: {format1}")
+    print(f"Path2 format: {format2}")
+
+    if format1 == "unknown":
+        print(f"Error: Unable to detect weight format in {args.path1}")
+        return 1
+
+    if format2 == "unknown":
+        print(f"Error: Unable to detect weight format in {args.path2}")
+        return 1
+
+    # Load weights based on format
+    print("\nLoading weights...")
+
+    if format1 == "safetensor":
+        weights1 = load_safetensor_weights(args.path1)
+    else:
+        weights1 = load_kt_weights(args.path1)
+
+    if format2 == "safetensor":
+        weights2 = load_safetensor_weights(args.path2)
+    else:
+        weights2 = load_kt_weights(args.path2)
+
+    # Compare weights
+    all_match, differences = compare_weights(weights1, weights2, args.tolerance)
+
+    return 0 if all_match else 1
+
+
+if __name__ == "__main__":
+    exit(main())
@@ -0,0 +1,488 @@
+#!/usr/bin/env python
+"""
+GPU Weight Quantization Tool for KTransformers
+
+This script quantizes model weights for CPU-GPU hybrid inference when integrating
+KTransformers with SGLang. It supports multiple quantization methods (GPTQ, RTN) and
+applies selective quantization to GPU-resident layers while preserving certain
+components (e.g., attention, gates, shared experts) in higher precision.
+
+Usage:
+    python convert_gpu_weights.py --model_id /path/to/model --output_dir /path/to/output --quant_method GPTQ --quant_type W4A16
+
+Example (GPTQ with calibration for best accuracy):
+    python convert_gpu_weights.py \
+        --model_id /mnt/data2/models/Qwen3-Next-80B-A3B-Instruct \
+        --output_dir /mnt/data2/models/Qwen3-Next-80B-A3B-Instruct-GPU-weight \
+        --quant_method GPTQ \
+        --quant_type W4A16
+
+Example (RTN for fast quantization without calibration):
+    python convert_gpu_weights.py \
+        --model_id /mnt/data/models/GLM-4.5-Air \
+        --output_dir /mnt/data/models/GLM-4.5-Air-GPU-weights-rtn \
+        --quant_method RTN \
+        --quant_type W4A16
+"""
+
+import os
+import sys
+import warnings
+import argparse
+
+# IMPORTANT: Parse force_cpu argument BEFORE importing torch
+# CUDA_VISIBLE_DEVICES must be set before torch initializes CUDA
+if __name__ == "__main__":
+    # Quick check for --force_cpu flag before full argument parsing
+    if "--force_cpu" in sys.argv:
+        os.environ["CUDA_VISIBLE_DEVICES"] = ""
+        warnings.filterwarnings("ignore", message="Can't initialize NVML")
+        print("🔧 Forced CPU-only mode (CUDA_VISIBLE_DEVICES set before torch import)")
+
+# Now it's safe to import torch and other GPU-dependent libraries
+import torch
+from accelerate import init_empty_weights, infer_auto_device_map
+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
+from llmcompressor import oneshot
+from llmcompressor.modifiers.quantization.gptq import GPTQModifier
+from llmcompressor.modifiers.quantization import QuantizationModifier
+from datasets import load_dataset
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="Quantize MoE models with selective quantization")
+
+    # Required arguments
+    parser.add_argument("--model_id", type=str, required=True, help="Path to the input model directory")
+    parser.add_argument("--output_dir", type=str, required=True, help="Path to save the quantized model")
+
+    # Optional arguments
+    parser.add_argument(
+        "--quant_method",
+        type=str,
+        choices=["GPTQ", "RTN"],
+        default="GPTQ",
+        help="Quantization method: GPTQ (calibration-based) or RTN (round-to-nearest, no calibration). Default: GPTQ",
+    )
+    parser.add_argument(
+        "--quant_type",
+        type=str,
+        choices=["W4A16", "W8A16"],
+        default="W8A16",
+        help="Quantization type: W4A16 (INT4) or W8A16 (INT8). Default: W8A16",
+    )
+    parser.add_argument(
+        "--num_calibration_samples",
+        type=int,
+        default=512,
+        help="Number of calibration samples (GPTQ only). Default: 512",
+    )
+    parser.add_argument(
+        "--max_sequence_length",
+        type=int,
+        default=2048,
+        help="Maximum sequence length for calibration (GPTQ only). Default: 2048",
+    )
+    parser.add_argument(
+        "--dampening_frac",
+        type=float,
+        default=0.1,
+        help="Dampening fraction to mitigate quantization noise (GPTQ only). Default: 0.1",
+    )
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        default="HuggingFaceH4/ultrachat_200k",
+        help="Dataset for calibration (GPTQ only). Default: HuggingFaceH4/ultrachat_200k",
+    )
+    parser.add_argument(
+        "--dataset_split", type=str, default="train_sft", help="Dataset split to use (GPTQ only). Default: train_sft"
+    )
+    parser.add_argument(
+        "--force_cpu", action="store_true", help="Force all computations to CPU (sets CUDA_VISIBLE_DEVICES='')"
+    )
+    parser.add_argument(
+        "--ignore_patterns",
+        type=str,
+        nargs="*",
+        default=[
+            "lm_head",
+            r"re:.*\.mlp\.gate$",
+            r"re:.*\.self_attn\..*$",
+            r"re:.*\.shared_expert\..*$",
+            r"re:.*\.shared_experts\..*$",
+            r"re:.*\.mlp\.shared_expert_gate$",
+            r"re:.*\.linear_attn\..*$",
+        ],
+        help="Regex patterns for layers to ignore during quantization",
+    )
+    parser.add_argument(
+        "--torch_dtype",
+        type=str,
+        choices=["bfloat16", "float16", "float32"],
+        default="bfloat16",
+        help="PyTorch dtype for model loading. Default: bfloat16",
+    )
+    parser.add_argument(
+        "--trust_remote_code", action="store_true", help="Allow loading of remote code (required for some models)"
+    )
+    parser.add_argument("--random_seed", type=int, default=42, help="Random seed for dataset shuffling. Default: 42")
+    parser.add_argument(
+        "--max_gpu_memory",
+        type=str,
+        default=None,
+        help="Maximum GPU memory for model weights per device (e.g., '40GiB'). "
+        "GPTQ quantization requires additional GPU memory for Hessian matrix computation, "
+        "so reserve 40-50%% of total VRAM. For example, use '40GiB' on 80GB GPUs. "
+        "Remaining layers will be offloaded to CPU. Default: use all available",
+    )
+    parser.add_argument(
+        "--max_cpu_memory",
+        type=str,
+        default=None,
+        help="Maximum CPU memory to use (e.g., '100GiB'). Default: use all available",
+    )
+
+    return parser.parse_args()
+
+
+def setup_environment(force_cpu=False):
+    """
+    Verify environment setup (actual setup happens before torch import).
+
+    Args:
+        force_cpu: If True, was requested to force CPU-only mode
+
+    Note:
+        CUDA_VISIBLE_DEVICES must be set BEFORE importing torch.
+        The actual environment setup is done at module import time.
+    """
+    if force_cpu:
+        # Verify the environment variable was set correctly
+        cuda_visible = os.environ.get("CUDA_VISIBLE_DEVICES", None)
+        if cuda_visible != "":
+            print("⚠️  Warning: force_cpu was requested but CUDA_VISIBLE_DEVICES is not empty")
+            print(f"   Current value: '{cuda_visible}'")
+            print("   This may happen if imported as a module. Recommend running as script.")
+        else:
+            print("✅ CPU-only mode verified (CUDA_VISIBLE_DEVICES is empty)")
+
+
+def get_torch_dtype(dtype_str):
+    """
+    Convert string to torch dtype.
+
+    Args:
+        dtype_str: String representation of dtype ("bfloat16", "float16", "float32")
+
+    Returns:
+        torch.dtype: Corresponding PyTorch dtype
+    """
+    dtype_map = {"bfloat16": torch.bfloat16, "float16": torch.float16, "float32": torch.float32}
+    return dtype_map[dtype_str]
+
+
+def check_dense_layers_and_update_ignore(model_id, ignore_patterns, trust_remote_code=False):
+    """
+    Check if the model has dense layers (first_k_dense_replace parameter) and add them to ignore list.
+
+    Some MoE models have dense MLP layers in the first few layers instead of MoE layers.
+    These dense layers should not be quantized using the same scheme as expert layers.
+
+    Args:
+        model_id: Path to the model
+        ignore_patterns: List of existing ignore patterns
+        trust_remote_code: Whether to trust remote code
+
+    Returns:
+        Updated ignore_patterns list with dense layer patterns added
+    """
+    print("🔍 Checking model configuration for dense layers...")
+
+    try:
+        # Load model configuration
+        config = AutoConfig.from_pretrained(model_id, trust_remote_code=trust_remote_code)
+
+        # Check if the model has first_k_dense_replace parameter
+        first_k_dense_replace = getattr(config, "first_k_dense_replace", None)
+
+        if first_k_dense_replace is not None and first_k_dense_replace > 0:
+            print(f"✅ Found dense layers configuration: first_k_dense_replace = {first_k_dense_replace}")
+            print(f"   Adding first {first_k_dense_replace} layers to ignore list...")
+
+            # Create regex pattern for dense layers (layers 0 to first_k_dense_replace-1)
+            if first_k_dense_replace == 1:
+                dense_pattern = r"re:model\.layers\.0\.mlp\..*$"
+            else:
+                # For multiple layers, use range pattern
+                layer_range = f"[0-{first_k_dense_replace-1}]"
+                dense_pattern = f"re:model\\.layers\\.{layer_range}\\.mlp\\..*$"
+
+            # Add the dense layer pattern to ignore list
+            updated_ignore_patterns = ignore_patterns + [dense_pattern]
+
+            print(f"   Dense layer pattern added: {dense_pattern}")
+            print(f"   This will ignore MLP components in layers 0-{first_k_dense_replace-1}")
+
+            return updated_ignore_patterns
+        else:
+            print("ℹ️  No dense layers detected (first_k_dense_replace not found or is 0)")
+            return ignore_patterns
+
+    except Exception as e:
+        print(f"⚠️  Warning: Could not check model config for dense layers: {e}")
+        print("   Proceeding with original ignore patterns...")
+        return ignore_patterns
+
+
+def load_and_prepare_dataset(dataset_name, dataset_split, num_samples, max_length, tokenizer, seed=42):
+    """
+    Load and prepare calibration dataset for GPTQ quantization.
+
+    GPTQ requires calibration data to compute optimal quantization parameters.
+    This function loads a conversation dataset, applies chat template, and tokenizes it.
+
+    Args:
+        dataset_name: HuggingFace dataset name
+        dataset_split: Dataset split to use (e.g., "train_sft")
+        num_samples: Number of samples to use for calibration
+        max_length: Maximum sequence length for tokenization
+        tokenizer: Model tokenizer
+        seed: Random seed for shuffling
+
+    Returns:
+        Dataset with tokenized calibration samples
+    """
+    print(f"📊 Loading dataset: {dataset_name}")
+
+    # Load dataset
+    ds = load_dataset(dataset_name, split=f"{dataset_split}[:{num_samples}]")
+    ds = ds.shuffle(seed=seed)
+
+    # Preprocess the data into the format the model is trained with
+    def preprocess(example):
+        return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
+
+    ds = ds.map(preprocess)
+
+    # Tokenize the data
+    def tokenize(sample):
+        return tokenizer(
+            sample["text"], padding=False, max_length=max_length, truncation=True, add_special_tokens=False
+        )
+
+    ds = ds.map(tokenize, remove_columns=ds.column_names)
+    print(f"✅ Dataset prepared with {len(ds)} samples")
+
+    return ds
+
+
+def main():
+    """
+    Main function for GPU weight quantization.
+
+    This performs weight quantization on model weights intended for GPU execution
+    in CPU-GPU hybrid inference scenarios. Supports two quantization methods:
+
+    1. GPTQ (default): Calibration-based quantization for better accuracy
+       - Requires calibration dataset
+       - Higher accuracy but slower
+       - Recommended for production use
+
+    2. RTN (Round-To-Nearest): Fast quantization without calibration
+       - No calibration dataset needed
+       - Faster but may have lower accuracy
+       - Good for quick testing or prototyping
+
+    The quantization is selective:
+    - Expert MLP weights are quantized to INT4/INT8
+    - Attention layers, gates, and shared experts remain in original precision
+    - Dense layers (if present) are excluded from quantization
+
+    The quantized model can be used with SGLang+KTransformers for heterogeneous
+    inference, where "hot" experts run on GPU and "cold" experts run on CPU.
+    """
+    args = parse_args()
+
+    # Setup environment
+    setup_environment(args.force_cpu)
+
+    # Convert torch dtype
+    torch_dtype = get_torch_dtype(args.torch_dtype)
+
+    print(f"🚀 Starting quantization process")
+    print(f"   Model: {args.model_id}")
+    print(f"   Output: {args.output_dir}")
+    print(f"   Quantization method: {args.quant_method}")
+    print(f"   Quantization type: {args.quant_type}")
+    if args.quant_method == "GPTQ":
+        print(f"   Calibration samples: {args.num_calibration_samples}")
+        print(f"   Max sequence length: {args.max_sequence_length}")
+    else:
+        print(f"   Calibration: Not required for {args.quant_method}")
+
+    # --------------------------------------------------------------------
+    # 0) Check for dense layers and update ignore patterns
+    # Dense layers in the first few layers should not be quantized
+    updated_ignore_patterns = check_dense_layers_and_update_ignore(
+        args.model_id, args.ignore_patterns, args.trust_remote_code
+    )
+
+    # --------------------------------------------------------------------
+    # 1) Build a dummy model (no weights) to infer a device map
+    # This determines optimal device placement for each module
+    if args.force_cpu:
+        # In force_cpu mode, directly get module names without calling infer_auto_device_map
+        # to avoid GPU memory allocation
+        print("🔍 Building CPU-only device map...")
+        with init_empty_weights():
+            dummy = AutoModelForCausalLM.from_pretrained(
+                args.model_id, torch_dtype=torch_dtype, trust_remote_code=args.trust_remote_code
+            )
+            device_map = {name: "cpu" for name, _ in dummy.named_modules() if name}
+            del dummy
+    else:
+        print("🔍 Inferring device map...")
+        with init_empty_weights():
+            dummy = AutoModelForCausalLM.from_pretrained(
+                args.model_id, torch_dtype=torch_dtype, trust_remote_code=args.trust_remote_code
+            )
+            # Build max_memory dict if specified
+            max_memory = None
+            if args.max_gpu_memory or args.max_cpu_memory:
+                max_memory = {}
+                if args.max_gpu_memory:
+                    # Apply to all available GPUs
+                    num_gpus = torch.cuda.device_count()
+                    for i in range(num_gpus):
+                        max_memory[i] = args.max_gpu_memory
+                    print(f"   GPU memory limit: {args.max_gpu_memory} per device ({num_gpus} GPUs)")
+
+                # Always set CPU memory when max_memory is used
+                # Otherwise infer_auto_device_map may trigger disk offloading
+                if args.max_cpu_memory:
+                    max_memory["cpu"] = args.max_cpu_memory
+                    print(f"   CPU memory limit: {args.max_cpu_memory}")
+                else:
+                    # Use a very large value to allow using all available CPU memory
+                    # This prevents disk offloading when user has enough RAM
+                    max_memory["cpu"] = "1000GiB"
+                    print(f"   CPU memory limit: 1000GiB (default, to prevent disk offloading)")
+
+            device_map = infer_auto_device_map(
+                dummy, no_split_module_classes=dummy._no_split_modules, max_memory=max_memory
+            )
+
+            # Check if disk offloading was triggered (not supported by llmcompressor)
+            disk_modules = [k for k, v in device_map.items() if v == "disk"]
+            if disk_modules:
+                print(f"❌ Error: {len(disk_modules)} modules would be offloaded to disk.")
+                print("   llmcompressor does not support disk offloading.")
+                print("   Solutions:")
+                print("   1. Increase --max_gpu_memory to use more GPU memory")
+                print("   2. Add --max_cpu_memory with higher value (e.g., '200GiB')")
+                print("   3. Ensure your machine has enough GPU + CPU memory")
+                raise RuntimeError(
+                    "Disk offloading is not supported by llmcompressor. "
+                    "Please ensure you have enough GPU + CPU memory."
+                )
+
+            del dummy
+    # --------------------------------------------------------------------
+    # 2) Load the full model weights with device mapping
+    # Note: offload_folder=None disables disk offloading (not supported by llmcompressor)
+    print("📥 Loading model...")
+    try:
+        model = AutoModelForCausalLM.from_pretrained(
+            args.model_id,
+            device_map=device_map,
+            torch_dtype=torch_dtype,
+            trust_remote_code=args.trust_remote_code,
+            offload_folder=None,  # Disable disk offloading (not supported by llmcompressor)
+        )
+    except Exception as e:
+        if "disk" in str(e).lower() or "offload" in str(e).lower():
+            print(f"❌ Error: Not enough GPU + CPU memory to load the model.")
+            print("   llmcompressor does not support disk offloading.")
+            print("   Solutions:")
+            print("   1. Increase --max_gpu_memory to use more GPU memory")
+            print("   2. Ensure you have enough CPU RAM for remaining layers")
+            print("   3. Use a machine with more memory")
+            raise
+        raise
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model_id)
+
+    # --------------------------------------------------------------------
+    # 3) Prepare calibration dataset
+    # GPTQ needs calibration data to compute optimal quantization parameters
+    if args.quant_method == "GPTQ":
+        ds = load_and_prepare_dataset(
+            args.dataset,
+            args.dataset_split,
+            args.num_calibration_samples,
+            args.max_sequence_length,
+            tokenizer,
+            args.random_seed,
+        )
+
+    # --------------------------------------------------------------------
+    # 4) Create quantization recipe with selective layer exclusion
+    print(f"⚙️  Setting up {args.quant_method} {args.quant_type} quantization recipe...")
+    if args.quant_method == "GPTQ":
+        # GPTQ: calibration-based quantization for better accuracy
+        recipe = GPTQModifier(
+            targets="Linear",  # Target all Linear layers
+            scheme=args.quant_type,  # W4A16 or W8A16
+            ignore=updated_ignore_patterns,  # Exclude specific patterns
+            dampening_frac=args.dampening_frac,
+        )
+    elif args.quant_method == "RTN":
+        # RTN (Round-To-Nearest): fast quantization without calibration
+        recipe = QuantizationModifier(
+            targets="Linear",  # Target all Linear layers
+            scheme=args.quant_type,  # W4A16 or W8A16
+            ignore=updated_ignore_patterns,  # Exclude specific patterns
+        )
+    else:
+        raise ValueError(f"Unsupported quantization method: {args.quant_method}")
+
+    print("🔧 Ignoring the following patterns from quantization:")
+    for i, pattern in enumerate(updated_ignore_patterns):
+        marker = "🆕" if i >= len(args.ignore_patterns) else "   "
+        print(f"   {marker} {pattern}")
+
+    # --------------------------------------------------------------------
+    # 5) Perform one-shot quantization
+    # GPTQ: calibration-based quantization to minimize accuracy loss
+    # RTN: fast round-to-nearest quantization without calibration
+    print("🎯 Starting one-shot quantization...")
+    if args.quant_method == "GPTQ":
+        # GPTQ requires calibration dataset
+        oneshot(
+            model=model,
+            dataset=ds,
+            recipe=recipe,
+            output_dir=args.output_dir,
+            max_seq_length=args.max_sequence_length,
+            num_calibration_samples=args.num_calibration_samples,
+        )
+    elif args.quant_method == "RTN":
+        # RTN does not require calibration dataset
+        oneshot(
+            model=model,
+            recipe=recipe,
+            output_dir=args.output_dir,
+        )
+    else:
+        raise ValueError(f"Unsupported quantization method: {args.quant_method}")
+
+    print(f"\n✅ Quantized model written to: {args.output_dir}")
+    print(f"   Quantization method: {args.quant_method}")
+    print(f"   Quantization type: {args.quant_type}")
+    print(f"   Ignored patterns remain in {args.torch_dtype}")
+    print("🎉 Quantization completed successfully!")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,98 @@
+import os
+import json
+from argparse import ArgumentParser
+from glob import glob
+from tqdm import tqdm
+
+import torch
+from safetensors.torch import load_file, save_file
+
+import gc
+
+
+def weight_dequant_cpu(x: torch.Tensor, s: torch.Tensor, block_size: int = 128) -> torch.Tensor:
+    assert x.dim() == 2 and s.dim() == 2, "Expect 2D tensors for x and s"
+    M, N = x.shape
+    n_m = (M + block_size - 1) // block_size
+    n_n = (N + block_size - 1) // block_size
+
+    y = torch.empty((M, N), dtype=torch.bfloat16, device="cpu")
+    for bm in range(n_m):
+        m0 = bm * block_size
+        m1 = min(m0 + block_size, M)
+        for bn in range(n_n):
+            n0 = bn * block_size
+            n1 = min(n0 + block_size, N)
+            scale = s[bm, bn].item()
+            sub = x[m0:m1, n0:n1].to(torch.float32) * scale
+            y[m0:m1, n0:n1] = sub.to(torch.bfloat16)
+    return y
+
+
+def main(fp8_path, bf16_path):
+    torch.set_default_dtype(torch.bfloat16)
+    os.makedirs(bf16_path, exist_ok=True)
+    model_index_file = os.path.join(fp8_path, "model.safetensors.index.json")
+    with open(model_index_file, "r") as f:
+        model_index = json.load(f)
+    weight_map = model_index["weight_map"]
+
+    loaded_files = {}
+    fp8_weight_names = []
+
+    def get_tensor(tensor_name):
+        file_name = weight_map[tensor_name]
+        if file_name not in loaded_files:
+            file_path = os.path.join(fp8_path, file_name)
+            loaded_files[file_name] = load_file(file_path, device="cpu")
+        return loaded_files[file_name][tensor_name]
+
+    safetensor_files = list(glob(os.path.join(fp8_path, "*.safetensors")))
+    safetensor_files.sort()
+    for safetensor_file in tqdm(safetensor_files, desc="weight file convert"):
+        file_name = os.path.basename(safetensor_file)
+        current_state_dict = load_file(safetensor_file, device="cpu")
+        loaded_files[file_name] = current_state_dict
+
+        new_state_dict = {}
+        for weight_name, weight in current_state_dict.items():
+            if weight_name.endswith("_scale_inv"):
+                continue
+            elif weight.element_size() == 1:
+                scale_inv_name = f"{weight_name}_scale_inv"
+                try:
+                    scale_inv = get_tensor(scale_inv_name)
+                    fp8_weight_names.append(weight_name)
+                    new_state_dict[weight_name] = weight_dequant_cpu(weight, scale_inv)
+                except KeyError:
+                    print(f"Warning: {weight_name}loss scale factor")
+                    new_state_dict[weight_name] = weight
+            else:
+                new_state_dict[weight_name] = weight
+
+        new_safetensor_file = os.path.join(bf16_path, file_name)
+        save_file(new_state_dict, new_safetensor_file)
+
+        if len(loaded_files) > 2:
+            oldest_file = next(iter(loaded_files))
+            del loaded_files[oldest_file]
+            gc.collect()
+            if torch.cuda.is_available():
+                torch.cuda.empty_cache()
+
+    new_model_index_file = os.path.join(bf16_path, "model.safetensors.index.json")
+    for weight_name in fp8_weight_names:
+        scale_inv_name = f"{weight_name}_scale_inv"
+        if scale_inv_name in weight_map:
+            weight_map.pop(scale_inv_name)
+    with open(new_model_index_file, "w") as f:
+        json.dump({"metadata": {}, "weight_map": weight_map}, f, indent=2)
+    print(f"Finish, Result in: {bf16_path}")
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser()
+    parser.add_argument("--input-fp8-hf-path", type=str, required=True, help="Kimi-K2 FP8 model")
+    parser.add_argument("--output-bf16-hf-path", type=str, required=True, help="BF16 model (After convert)")
+    args = parser.parse_args()
+    main(args.input_fp8_hf_path, args.output_bf16_hf_path)
@@ -0,0 +1,477 @@
+#!/usr/bin/env python3
+"""Convert KT fused expert LoRA checkpoints into an SGLang adapter directory."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import shutil
+from pathlib import Path
+from typing import Dict, Iterable, Mapping
+
+import torch
+from safetensors.torch import load_file, save_file
+
+
+FUSED_EXPERT_LORA_FILE = "fused_expert_lora.safetensors"
+ADAPTER_MODEL_FILE = "adapter_model.safetensors"
+ADAPTER_CONFIG_FILE = "adapter_config.json"
+
+KT_NAME_MAP = {
+    "gate_lora_a": ("gate_proj", "lora_A", 1),
+    "gate_lora_b": ("gate_proj", "lora_B", 2),
+    "up_lora_a": ("up_proj", "lora_A", 1),
+    "up_lora_b": ("up_proj", "lora_B", 2),
+    "down_lora_a": ("down_proj", "lora_A", 1),
+    "down_lora_b": ("down_proj", "lora_B", 2),
+}
+
+TARGET_MODULE_ORDER = [
+    "q_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "up_proj",
+    "down_proj",
+    "in_proj_qkv",
+    "in_proj_z",
+    "in_proj_b",
+    "in_proj_a",
+    "out_proj",
+    "embed_tokens",
+    "lm_head",
+]
+
+KT_FUSED_KEY_RE = re.compile(r"^layers\.(\d+)\.experts\.([^.]+)$")
+
+
+def _load_json(path: Path) -> dict:
+    with path.open("r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def _write_json(path: Path, data: Mapping) -> None:
+    with path.open("w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2, sort_keys=True)
+        f.write("\n")
+
+
+def _clean_adapter_key(key: str) -> str:
+    """Match the existing SGLang converter's PEFT key cleanup."""
+    key = key.replace("base_model.model.", "")
+    key = key.replace(".orig_module", "")
+    return key
+
+
+def _ordered_target_modules(modules: Iterable[str]) -> list[str]:
+    seen = set(modules)
+    ordered = [name for name in TARGET_MODULE_ORDER if name in seen]
+    ordered.extend(sorted(seen.difference(ordered)))
+    return ordered
+
+
+def _infer_target_module_from_key(key: str) -> str | None:
+    if "lora_embedding_A" in key or "lora_embedding_B" in key:
+        if "embed_tokens" in key:
+            return "embed_tokens"
+        if "lm_head" in key or "unembed_tokens" in key:
+            return "lm_head"
+
+    marker = ".lora_"
+    if marker not in key:
+        return None
+    prefix = key.split(marker, 1)[0]
+    if "." not in prefix:
+        return prefix
+    return prefix.rsplit(".", 1)[-1]
+
+
+def _merge_tensor(tensors: Dict[str, torch.Tensor], key: str, value: torch.Tensor) -> None:
+    if key in tensors:
+        raise ValueError(f"Duplicate output tensor key: {key}")
+    tensors[key] = value.detach().cpu()
+
+
+def _load_existing_adapter(input_dir: Path) -> tuple[dict[str, torch.Tensor], set[str]]:
+    adapter_path = input_dir / ADAPTER_MODEL_FILE
+    if not adapter_path.exists():
+        return {}, set()
+
+    tensors: dict[str, torch.Tensor] = {}
+    target_modules: set[str] = set()
+    for key, value in load_file(str(adapter_path)).items():
+        cleaned_key = _clean_adapter_key(key)
+        _merge_tensor(tensors, cleaned_key, value)
+        target_module = _infer_target_module_from_key(cleaned_key)
+        if target_module is not None:
+            target_modules.add(target_module)
+    return tensors, target_modules
+
+
+def _convert_fused_expert_lora(
+    fused_path: Path,
+) -> tuple[dict[str, torch.Tensor], int, set[str]]:
+    if not fused_path.exists():
+        raise FileNotFoundError(f"Missing {FUSED_EXPERT_LORA_FILE}: {fused_path}")
+
+    output: dict[str, torch.Tensor] = {}
+    ranks: set[int] = set()
+    expert_counts: set[int] = set()
+    target_modules: set[str] = set()
+
+    for key, tensor in sorted(load_file(str(fused_path)).items()):
+        match = KT_FUSED_KEY_RE.match(key)
+        if match is None:
+            raise ValueError(f"Unexpected key in {FUSED_EXPERT_LORA_FILE}: {key}")
+
+        layer_idx, kt_name = match.groups()
+        if kt_name not in KT_NAME_MAP:
+            raise ValueError(f"Unsupported KT fused expert LoRA tensor: {key}")
+        if tensor.dim() != 3:
+            raise ValueError(f"{key} must be 3D [E, ...], got shape {tuple(tensor.shape)}")
+
+        proj_name, lora_name, rank_dim = KT_NAME_MAP[kt_name]
+        expert_count = int(tensor.shape[0])
+        rank = int(tensor.shape[rank_dim])
+        expert_counts.add(expert_count)
+        ranks.add(rank)
+        target_modules.add(proj_name)
+
+        for expert_idx in range(expert_count):
+            output_key = (
+                f"model.layers.{layer_idx}.mlp.experts.{expert_idx}."
+                f"{proj_name}.{lora_name}.weight"
+            )
+            _merge_tensor(output, output_key, tensor[expert_idx].contiguous())
+
+    if not output:
+        raise ValueError(f"No tensors found in {fused_path}")
+    if len(expert_counts) != 1:
+        raise ValueError(f"Inconsistent expert counts in {FUSED_EXPERT_LORA_FILE}: {sorted(expert_counts)}")
+    if len(ranks) != 1:
+        raise ValueError(f"Inconsistent LoRA ranks in {FUSED_EXPERT_LORA_FILE}: {sorted(ranks)}")
+
+    return output, next(iter(ranks)), target_modules
+
+
+def _build_adapter_config(
+    input_dir: Path,
+    rank: int,
+    target_modules: set[str],
+    base_model_name_or_path: str,
+    lora_alpha: float | None,
+    *,
+    include_input_target_modules: bool = True,
+) -> dict:
+    config_path = input_dir / ADAPTER_CONFIG_FILE
+    config = _load_json(config_path) if config_path.exists() else {}
+
+    if "lora_alpha" in config:
+        final_alpha = config["lora_alpha"]
+    elif lora_alpha is not None:
+        final_alpha = lora_alpha
+    else:
+        raise ValueError(
+            f"No {ADAPTER_CONFIG_FILE} with lora_alpha found in {input_dir}; "
+            "pass --lora-alpha to preserve runtime scaling."
+        )
+
+    existing_targets = config.get("target_modules", [])
+    if include_input_target_modules and isinstance(existing_targets, list):
+        target_modules.update(str(name).split(".")[-1] for name in existing_targets)
+
+    config["peft_type"] = config.get("peft_type", "LORA")
+    config["r"] = rank
+    config["lora_alpha"] = final_alpha
+    config["target_modules"] = _ordered_target_modules(target_modules)
+    config["bias"] = config.get("bias", "none")
+    config["task_type"] = config.get("task_type", "CAUSAL_LM")
+    config["base_model_name_or_path"] = base_model_name_or_path
+
+    return config
+
+
+def _paths_have_ancestor_relationship(left: Path, right: Path) -> bool:
+    if left == right:
+        return True
+    try:
+        left.relative_to(right)
+        return True
+    except ValueError:
+        pass
+    try:
+        right.relative_to(left)
+        return True
+    except ValueError:
+        return False
+
+
+def _validate_no_ancestor_paths(
+    paths: Iterable[Path],
+    *,
+    label: str,
+) -> None:
+    resolved = list(paths)
+    for i, left in enumerate(resolved):
+        for right in resolved[i + 1 :]:
+            if _paths_have_ancestor_relationship(left, right):
+                raise ValueError(
+                    f"{label} cannot have ancestor/descendant relationships: "
+                    f"{left} and {right}."
+                )
+
+
+def _prepare_output_dir(output_path: Path, input_path: Path, overwrite: bool) -> None:
+    _validate_output_dir(output_path, input_path, overwrite)
+    if output_path.exists() and any(output_path.iterdir()):
+        shutil.rmtree(output_path)
+    output_path.mkdir(parents=True, exist_ok=True)
+
+
+def _validate_output_dir(output_path: Path, input_path: Path, overwrite: bool) -> None:
+    if output_path == input_path:
+        raise ValueError("Output directory must be different from input directory.")
+    if _paths_have_ancestor_relationship(output_path, input_path):
+        raise ValueError(
+            "Output and input directories cannot be ancestor/descendant of each other: "
+            f"output={output_path}, input={input_path}."
+        )
+    if output_path.exists() and not output_path.is_dir():
+        raise FileExistsError(f"Output path exists and is not a directory: {output_path}")
+    if output_path.exists() and any(output_path.iterdir()):
+        if not overwrite:
+            raise FileExistsError(f"Output directory is not empty: {output_path}")
+
+
+def _infer_lora_rank_from_tensor(key: str, tensor: torch.Tensor) -> int | None:
+    if ".lora_A." in key:
+        return int(tensor.shape[0])
+    if ".lora_B." in key:
+        return int(tensor.shape[1])
+    return None
+
+
+def _validate_nonexpert_rank(
+    existing_tensors: Mapping[str, torch.Tensor],
+    expert_rank: int,
+    input_dir: Path,
+) -> None:
+    if not existing_tensors:
+        return
+
+    config_path = input_dir / ADAPTER_CONFIG_FILE
+    if config_path.exists():
+        config_rank = _load_json(config_path).get("r")
+        if config_rank is not None and int(config_rank) != expert_rank:
+            raise ValueError(
+                f"Non-expert adapter rank mismatch: adapter_config.json r={config_rank}, "
+                f"but fused expert LoRA rank={expert_rank}."
+            )
+
+    for key, tensor in existing_tensors.items():
+        tensor_rank = _infer_lora_rank_from_tensor(key, tensor)
+        if tensor_rank is None:
+            continue
+        if tensor_rank != expert_rank:
+            raise ValueError(
+                f"Non-expert adapter tensor rank mismatch for {key}: "
+                f"tensor rank={tensor_rank}, fused expert LoRA rank={expert_rank}."
+            )
+
+
+def _write_adapter(
+    output_path: Path,
+    input_path: Path,
+    tensors: dict[str, torch.Tensor],
+    config: Mapping,
+    *,
+    overwrite: bool,
+) -> None:
+    _prepare_output_dir(output_path, input_path, overwrite)
+    save_file(tensors, str(output_path / ADAPTER_MODEL_FILE), metadata={"format": "pt"})
+    _write_json(output_path / ADAPTER_CONFIG_FILE, config)
+
+
+def convert_kt_to_sglang_adapter(
+    input_dir: str | os.PathLike,
+    output_dir: str | os.PathLike,
+    *,
+    base_model_name_or_path: str,
+    lora_alpha: float | None = None,
+    overwrite: bool = False,
+    expert_output_dir: str | os.PathLike | None = None,
+    nonexpert_output_dir: str | os.PathLike | None = None,
+) -> dict:
+    input_path = Path(input_dir).expanduser().resolve()
+    output_path = Path(output_dir).expanduser().resolve()
+    expert_output_path = (
+        Path(expert_output_dir).expanduser().resolve()
+        if expert_output_dir is not None
+        else None
+    )
+    nonexpert_output_path = (
+        Path(nonexpert_output_dir).expanduser().resolve()
+        if nonexpert_output_dir is not None
+        else None
+    )
+
+    if not input_path.is_dir():
+        raise FileNotFoundError(f"Input directory not found: {input_path}")
+
+    output_paths = [output_path]
+    output_paths.extend(path for path in (expert_output_path, nonexpert_output_path) if path is not None)
+    if len(set(output_paths)) != len(output_paths):
+        raise ValueError("Merged, expert, and non-expert output directories must be distinct.")
+    _validate_no_ancestor_paths(
+        output_paths,
+        label="Merged/expert/non-expert output directories",
+    )
+    for path in output_paths:
+        _validate_output_dir(path, input_path, overwrite)
+
+    existing_tensors, existing_targets = _load_existing_adapter(input_path)
+    fused_tensors, rank, fused_targets = _convert_fused_expert_lora(input_path / FUSED_EXPERT_LORA_FILE)
+    _validate_nonexpert_rank(existing_tensors, rank, input_path)
+    if nonexpert_output_path is not None and not existing_tensors:
+        raise ValueError(
+            f"Cannot write non-expert adapter: no {ADAPTER_MODEL_FILE} found in {input_path}."
+        )
+
+    tensors: dict[str, torch.Tensor] = {}
+    for key, value in existing_tensors.items():
+        _merge_tensor(tensors, key, value)
+    for key, value in fused_tensors.items():
+        _merge_tensor(tensors, key, value)
+
+    target_modules = set(existing_targets)
+    target_modules.update(fused_targets)
+    config = _build_adapter_config(
+        input_path,
+        rank,
+        target_modules,
+        base_model_name_or_path,
+        lora_alpha,
+    )
+
+    _write_adapter(output_path, input_path, tensors, config, overwrite=overwrite)
+
+    split_outputs: dict[str, dict] = {}
+    if expert_output_path is not None:
+        expert_config = _build_adapter_config(
+            input_path,
+            rank,
+            set(fused_targets),
+            base_model_name_or_path,
+            lora_alpha,
+            include_input_target_modules=False,
+        )
+        _write_adapter(
+            expert_output_path,
+            input_path,
+            fused_tensors,
+            expert_config,
+            overwrite=overwrite,
+        )
+        split_outputs["expert"] = {
+            "output_dir": str(expert_output_path),
+            "tensor_count": len(fused_tensors),
+            "target_modules": expert_config["target_modules"],
+        }
+
+    if nonexpert_output_path is not None:
+        nonexpert_config = _build_adapter_config(
+            input_path,
+            rank,
+            set(existing_targets),
+            base_model_name_or_path,
+            lora_alpha,
+            include_input_target_modules=False,
+        )
+        _write_adapter(
+            nonexpert_output_path,
+            input_path,
+            existing_tensors,
+            nonexpert_config,
+            overwrite=overwrite,
+        )
+        split_outputs["nonexpert"] = {
+            "output_dir": str(nonexpert_output_path),
+            "tensor_count": len(existing_tensors),
+            "target_modules": nonexpert_config["target_modules"],
+        }
+
+    return {
+        "input_dir": str(input_path),
+        "output_dir": str(output_path),
+        "tensor_count": len(tensors),
+        "rank": rank,
+        "target_modules": config["target_modules"],
+        "lora_alpha": config["lora_alpha"],
+        "split_outputs": split_outputs,
+    }
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Convert KT fused expert LoRA weights to an SGLang adapter directory."
+    )
+    parser.add_argument("input_dir", help="Directory containing fused_expert_lora.safetensors.")
+    parser.add_argument("output_dir", help="Destination adapter directory.")
+    parser.add_argument(
+        "--base-model-name-or-path",
+        required=True,
+        help="Base model path/name to write into adapter_config.json.",
+    )
+    parser.add_argument(
+        "--lora-alpha",
+        type=float,
+        default=None,
+        help="LoRA alpha to use when input adapter_config.json is absent.",
+    )
+    parser.add_argument(
+        "--overwrite",
+        action="store_true",
+        help="Remove and recreate output_dir if it already contains files.",
+    )
+    parser.add_argument(
+        "--expert-output-dir",
+        default=None,
+        help="Optional destination for a split expert-only adapter directory.",
+    )
+    parser.add_argument(
+        "--nonexpert-output-dir",
+        default=None,
+        help="Optional destination for a split non-expert-only adapter directory.",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    summary = convert_kt_to_sglang_adapter(
+        args.input_dir,
+        args.output_dir,
+        base_model_name_or_path=args.base_model_name_or_path,
+        lora_alpha=args.lora_alpha,
+        overwrite=args.overwrite,
+        expert_output_dir=args.expert_output_dir,
+        nonexpert_output_dir=args.nonexpert_output_dir,
+    )
+    print(
+        "Converted KT fused expert LoRA adapter: "
+        f"{summary['tensor_count']} tensors, rank={summary['rank']}, "
+        f"target_modules={summary['target_modules']}"
+    )
+    for name, split_summary in summary["split_outputs"].items():
+        print(
+            f"Wrote {name} adapter: {split_summary['tensor_count']} tensors, "
+            f"target_modules={split_summary['target_modules']}, "
+            f"output_dir={split_summary['output_dir']}"
+        )
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,193 @@
+import argparse
+import json
+import os
+from collections import defaultdict
+from typing import Dict, Iterable, List, Optional, Tuple
+
+import torch
+from safetensors.torch import save_file, safe_open
+
+from compressed_tensors.compressors import unpack_from_int32
+
+
+def _load_config(model_dir: str, config_path: Optional[str]) -> Tuple[int, int, int]:
+    cfg_path = config_path or os.path.join(model_dir, "config.json")
+    with open(cfg_path, "r") as f:
+        cfg = json.load(f)
+    hidden_size = int(cfg.get("hidden_size"))
+    inter_size = int(cfg.get("moe_intermediate_size"))
+    group_size = int(
+        cfg.get("quantization_config", {})
+        .get("config_groups", {})
+        .get("group_0", {})
+        .get("weights", {})
+        .get("group_size", 32)
+    )
+    return hidden_size, inter_size, group_size
+
+
+def _dequantize_tensor(
+    weight_packed: torch.Tensor,
+    weight_scale: torch.Tensor,
+    weight_shape: torch.Tensor,
+    group_size: int,
+) -> torch.Tensor:
+    if isinstance(weight_shape, torch.Tensor):
+        shape = tuple(int(v) for v in weight_shape.view(-1).tolist())
+    else:
+        shape = tuple(weight_shape)
+    weight = unpack_from_int32(weight_packed, 4, shape)
+    if group_size > 0:
+        scale = weight_scale.to(torch.float32)
+        if scale.dim() == 1:
+            scale = scale.unsqueeze(1)
+        scales = torch.repeat_interleave(scale, repeats=group_size, dim=1)
+    else:
+        scales = weight_scale.to(torch.float32)
+    if scales.shape != weight.shape:
+        if scales.numel() == weight.numel():
+            scales = scales.reshape_as(weight)
+        else:
+            raise ValueError(f"Scale shape {scales.shape} incompatible with weight shape {weight.shape}")
+    bf16 = (weight.to(torch.float32) * scales).to(torch.bfloat16)
+    return bf16.contiguous()
+
+
+def _is_quantized_weight_key(key: str) -> bool:
+    if ".mlp.experts." not in key or ".shared_experts." in key:
+        return False
+    suffixes = ("weight_packed", "weight_scale", "weight_shape")
+    for proj in ("gate_proj", "up_proj", "down_proj"):
+        for suffix in suffixes:
+            if key.endswith(f".{proj}.{suffix}"):
+                return True
+    return False
+
+
+def convert_file(
+    input_path: str,
+    output_path: str,
+    group_size: int,
+    skip_existing: bool = True,
+):
+    if skip_existing and os.path.exists(output_path):
+        print(f"[skip] {output_path} already exists.")
+        return
+
+    tensors: Dict[str, torch.Tensor] = {}
+    expert_buffers: Dict[str, Dict[str, Dict[str, torch.Tensor]]] = defaultdict(lambda: defaultdict(dict))
+
+    with safe_open(input_path, framework="pt") as reader:
+        keys = list(reader.keys())
+        for key in keys:
+            tensor = reader.get_tensor(key).detach().cpu()
+
+            if not _is_quantized_weight_key(key):
+                tensors[key] = tensor
+                continue
+
+            parts = key.split(".")
+            try:
+                expert_idx = parts.index("experts")
+            except ValueError:
+                tensors[key] = tensor
+                continue
+
+            prefix = ".".join(parts[: expert_idx + 2])
+            project = parts[-2]
+            suffix = parts[-1]
+            expert_buffers[prefix][project][suffix] = tensor
+
+    stats = {
+        "converted": 0,
+        "skipped": 0,
+    }
+
+    for prefix, components in expert_buffers.items():
+        for proj_name in ["gate_proj", "up_proj", "down_proj"]:
+            proj_data = components.get(proj_name, {})
+            required = {"weight_packed", "weight_scale", "weight_shape"}
+            if not required.issubset(proj_data.keys()):
+                print(f"[warn] Missing components for {prefix}.{proj_name}, keeping quantized tensors.")
+                for suffix, value in proj_data.items():
+                    tensors[f"{prefix}.{proj_name}.{suffix}"] = value
+                stats["skipped"] += 1
+                continue
+
+            bf16_weight = _dequantize_tensor(
+                proj_data["weight_packed"].to(torch.int32),
+                proj_data["weight_scale"].to(torch.float32),
+                proj_data["weight_shape"],
+                group_size,
+            )
+            tensors[f"{prefix}.{proj_name}.weight"] = bf16_weight.to(torch.bfloat16)
+            stats["converted"] += 1
+            print(f"    converted {prefix}.{proj_name}.weight -> bf16")
+
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    save_file(tensors, output_path)
+    print(f"[done] wrote {output_path} (converted={stats['converted']}, skipped={stats['skipped']})")
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="Convert MoE experts to BF16 weights.")
+    parser.add_argument("--model-dir", required=True, help="Directory containing safetensors checkpoints.")
+    parser.add_argument(
+        "--output-dir",
+        default=None,
+        help="Destination directory for converted checkpoints (default: <model-dir>_bf16).",
+    )
+    parser.add_argument(
+        "--files",
+        nargs="+",
+        default=None,
+        help="Specific safetensor filenames to convert (relative to model-dir). Convert all if omitted.",
+    )
+    parser.add_argument(
+        "--config-path",
+        default=None,
+        help="Path to config.json for extracting group_size (default: model-dir/config.json).",
+    )
+    parser.add_argument(
+        "--overwrite",
+        action="store_true",
+        help="Rewrite output files even if they already exist.",
+    )
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    model_dir = os.path.abspath(args.model_dir)
+    output_dir = os.path.abspath(args.output_dir or f"{model_dir}_bf16")
+
+    if not os.path.isdir(model_dir):
+        raise FileNotFoundError(f"Model directory not found: {model_dir}")
+
+    _, _, group_size = _load_config(model_dir, args.config_path)
+
+    if args.files:
+        targets = [os.path.join(model_dir, fname) for fname in args.files]
+    else:
+        targets = [
+            os.path.join(model_dir, name) for name in sorted(os.listdir(model_dir)) if name.endswith(".safetensors")
+        ]
+
+    if not targets:
+        print("No safetensors checkpoints found.")
+        return
+
+    total = len(targets)
+
+    for idx, path in enumerate(targets, start=1):
+        if not os.path.isfile(path):
+            print(f"[skip] {path} is not a file.")
+            continue
+        rel = os.path.relpath(path, model_dir)
+        output_path = os.path.join(output_dir, rel)
+        print(f"[{idx}/{total}] converting {rel}")
+        convert_file(path, output_path, group_size, skip_existing=not args.overwrite)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,57 @@
+#!/usr/bin/env sh
+# Install git hooks from kt-kernel/.githooks into the monorepo's .git/hooks by
+# creating symlinks (or copying if symlink fails).
+
+set -eu
+
+# This script lives in kt-kernel/scripts/, so REPO_ROOT = kt-kernel
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+HOOKS_SRC="$REPO_ROOT/.githooks"
+
+# Detect the top-level Git worktree (the monorepo root: ktransformers)
+GIT_TOP="$(git rev-parse --show-toplevel 2>/dev/null || true)"
+if [ -z "$GIT_TOP" ] || [ ! -d "$GIT_TOP/.git" ]; then
+  echo "[install-git-hooks] Not inside a git worktree; skipping hooks installation." >&2
+  exit 0
+fi
+
+GIT_DIR="$GIT_TOP/.git"
+HOOKS_DEST="$GIT_DIR/hooks"
+
+if [ ! -d "$HOOKS_SRC" ]; then
+  echo "[install-git-hooks] No .githooks directory found at $HOOKS_SRC" >&2
+  exit 1
+fi
+
+echo "[install-git-hooks] Installing git hooks from $HOOKS_SRC to $HOOKS_DEST (repo: $GIT_TOP)"
+
+# Ensure all source hook files are executable so that even if copied (not symlinked) they run.
+for src_hook in "$HOOKS_SRC"/*; do
+  [ -f "$src_hook" ] || continue
+  if [ ! -x "$src_hook" ]; then
+    chmod +x "$src_hook" || true
+  fi
+done
+
+for hook in "$HOOKS_SRC"/*; do
+  [ -e "$hook" ] || continue
+  name=$(basename "$hook")
+  dest="$HOOKS_DEST/$name"
+
+  # Remove existing hook if it's our symlink or a file
+  if [ -L "$dest" ] || [ -f "$dest" ]; then
+    rm -f "$dest"
+  fi
+
+  # Try symlink first
+  if ln -s "$hook" "$dest" 2>/dev/null; then
+    echo "linked $name"
+  else
+    # Fall back to copying and preserve executable bit
+    cp "$hook" "$dest"
+    chmod +x "$dest"
+    echo "copied $name"
+  fi
+done
+
+echo "[install-git-hooks] Done. Hooks installed."
@@ -0,0 +1,278 @@
+#!/usr/bin/env python3
+
+import argparse
+import os
+import glob
+import numpy as np
+import torch
+from safetensors.torch import save_file
+import gc
+import json
+import shutil
+import sys
+
+
+def discover_layers(input_path: str):
+    """Discover all layer folders in the input directory."""
+    layer_folders = []
+    for item in os.listdir(input_path):
+        if item.startswith("_layer_"):
+            try:
+                layer_idx = int(item.split("_")[-1])
+                layer_folders.append((layer_idx, item))
+            except ValueError:
+                continue
+    layer_folders.sort(key=lambda x: x[0])
+    return layer_folders
+
+
+def discover_numa_folders(layer_path: str):
+    """Discover all NUMA folders within a layer folder."""
+    numa_folders = []
+    for item in os.listdir(layer_path):
+        if item.startswith("_numa_"):
+            try:
+                numa_idx = int(item.split("_")[-1])
+                numa_folders.append((numa_idx, item))
+            except ValueError:
+                continue
+    numa_folders.sort(key=lambda x: x[0])
+    return numa_folders
+
+
+def detect_quant_method(layer_path: str):
+    """Detect quantization method from file names (INT4 vs INT8)."""
+    for root, _, files in os.walk(layer_path):
+        for f in files:
+            if f.startswith("MOE_INT4_"):
+                return "moe_int4", "MOE_INT4"
+            elif f.startswith("MOE_INT8_"):
+                return "moe_int8", "MOE_INT8"
+            elif f.startswith("INT4_"):
+                return "int4", "INT4"
+            elif f.startswith("INT8_"):
+                return "int8", "INT8"
+    raise ValueError(f"Could not detect quant method in {layer_path}")
+
+
+def load_binary_tensor(file_path: str) -> torch.Tensor:
+    """Load .kt format binary tensor file."""
+    if not os.path.exists(file_path):
+        raise FileNotFoundError(f"File not found: {file_path}")
+
+    with open(file_path, "rb") as f:
+        binary_data = f.read()
+
+    if "scale" in file_path:
+        np_array = np.frombuffer(binary_data, dtype=np.float32)
+    else:
+        np_array = np.frombuffer(binary_data, dtype=np.int8)
+
+    return torch.from_numpy(np_array.copy())
+
+
+def process_layer(layer_path: str, amx_prefix: str, layer_idx: int) -> dict:
+    """Process a single layer folder and return all tensors."""
+    tensors = {}
+    numa_folders = discover_numa_folders(layer_path)
+
+    if not numa_folders:
+        print(f"  Warning: No NUMA folders found in {layer_path}", file=sys.stderr)
+        return tensors
+
+    proj_mappings = [
+        ("down", "ffn_down_exps"),
+        ("gate", "ffn_gate_exps"),
+        ("up", "ffn_up_exps"),
+    ]
+
+    for numa_idx, numa_folder in numa_folders:
+        numa_path = os.path.join(layer_path, numa_folder)
+
+        for proj_name, proj_key in proj_mappings:
+            quant_pattern = os.path.join(numa_path, f"{amx_prefix}_{proj_name}_*Byte_quant_.kt")
+            scale_pattern = os.path.join(numa_path, f"{amx_prefix}_{proj_name}_*Byte_scale_.kt")
+
+            quant_files = sorted(glob.glob(quant_pattern))
+            scale_files = sorted(glob.glob(scale_pattern))
+
+            for quant_file in quant_files:
+                filename = os.path.basename(quant_file)
+                remainder = filename[len(f"{amx_prefix}_{proj_name}_"):]
+                try:
+                    expert_idx = int(remainder.split("_")[0])
+                except (ValueError, IndexError):
+                    print(f"    Warning: Could not parse expert index from {filename}", file=sys.stderr)
+                    continue
+
+                weight_key = f"blk.{layer_idx}.{proj_key}.{expert_idx}.numa.{numa_idx}.weight"
+                tensors[weight_key] = load_binary_tensor(quant_file)
+
+            for scale_file in scale_files:
+                filename = os.path.basename(scale_file)
+                remainder = filename[len(f"{amx_prefix}_{proj_name}_"):]
+                try:
+                    expert_idx = int(remainder.split("_")[0])
+                except (ValueError, IndexError):
+                    print(f"    Warning: Could not parse expert index from {filename}", file=sys.stderr)
+                    continue
+
+                scale_key = f"blk.{layer_idx}.{proj_key}.{expert_idx}.numa.{numa_idx}.scale"
+                tensors[scale_key] = load_binary_tensor(scale_file)
+
+    return tensors
+
+
+def write_shards(accumulated_tensors: dict, output_path: str, shard_counter: dict, keep_remainder: bool = True):
+    """Write accumulated tensors to one or more shard files.
+    
+    Args:
+        accumulated_tensors: Dict of tensors to write
+        output_path: Output directory
+        shard_counter: Dict with 'shard' and 'max_tensors' keys
+        keep_remainder: If True, keep leftover tensors in accumulator for next batch
+    """
+    if not accumulated_tensors:
+        return
+
+    max_tensors = shard_counter["max_tensors"]
+    current_shard = shard_counter["shard"]
+    total_tensors = len(accumulated_tensors)
+
+    if total_tensors <= max_tensors:
+        if not keep_remainder:
+            output_file = os.path.join(output_path, f"model-{current_shard:05d}.safetensors")
+            save_file(accumulated_tensors, output_file)
+            print(f"  Saved {total_tensors} tensors to {output_file}")
+            shard_counter["shard"] = current_shard + 1
+            accumulated_tensors.clear()
+        else:
+            pass  # Keep accumulating until we hit max_tensors
+    else:
+        full_shards = total_tensors // max_tensors
+        remainder = total_tensors % max_tensors
+        
+        items = list(accumulated_tensors.items())
+        
+        # Write full shards
+        for i in range(full_shards):
+            batch = dict(items[i * max_tensors : (i + 1) * max_tensors])
+            output_file = os.path.join(output_path, f"model-{current_shard:05d}.safetensors")
+            save_file(batch, output_file)
+            print(f"  Saved {len(batch)} tensors to {output_file}")
+            current_shard += 1
+        
+        # Keep remainder for next batch if enabled
+        if keep_remainder and remainder > 0:
+            remainder_items = dict(items[full_shards * max_tensors:])
+            accumulated_tensors.clear()
+            accumulated_tensors.update(remainder_items)
+            print(f"  Rolled over {remainder} tensors to next batch")
+        elif remainder > 0:
+            # Write remainder as final shard
+            batch = dict(items[full_shards * max_tensors:])
+            output_file = os.path.join(output_path, f"model-{current_shard:05d}.safetensors")
+            save_file(batch, output_file)
+            print(f"  Saved {len(batch)} tensors to {output_file}")
+            current_shard += 1
+            accumulated_tensors.clear()
+        
+        shard_counter["shard"] = current_shard
+
+
+def copy_config_files(original_path: str, output_path: str):
+    """Copy config and tokenizer files from original model folder."""
+    config_files = [
+        "config.json",
+        "tokenizer.json",
+        "tokenizer_config.json",
+        "special_tokens_map.json",
+    ]
+
+    for config_file in config_files:
+        src_path = os.path.join(original_path, config_file)
+        if os.path.exists(src_path):
+            dst_path = os.path.join(output_path, config_file)
+            shutil.copy2(src_path, dst_path)
+            print(f"Copied: {config_file}")
+        else:
+            print(f"Warning: {config_file} not found in {original_path}, skipping", file=sys.stderr)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Merge CPU-optimized weights from nested folder structure to sharded safetensors"
+    )
+    parser.add_argument(
+        "--input-path", "-i", required=True, help="Input directory with nested _layer_* folders"
+    )
+    parser.add_argument("--output", "-o", required=True, help="Output directory for merged safetensors")
+    parser.add_argument(
+        "--original-path",
+        "-r",
+        default=None,
+        help="Original model folder with config.json and tokenizer files to copy",
+    )
+    parser.add_argument(
+        "--max-tensors",
+        type=int,
+        default=3000,
+        help="Maximum tensors per safetensors shard (default: 3000)",
+    )
+
+    args = parser.parse_args()
+
+    if not os.path.exists(args.input_path):
+        print(f"Error: Input path does not exist: {args.input_path}", file=sys.stderr)
+        return 1
+
+    os.makedirs(args.output, exist_ok=True)
+
+    print("Discovering layer folders...")
+    layer_folders = discover_layers(args.input_path)
+    if not layer_folders:
+        print(f"Error: No _layer_* folders found in {args.input_path}", file=sys.stderr)
+        return 1
+
+    print(f"Found {len(layer_folders)} layer folders")
+
+    print("Detecting quantization method...")
+    first_layer_path = os.path.join(args.input_path, layer_folders[0][1])
+    quant_method, amx_prefix = detect_quant_method(first_layer_path)
+    print(f"Detected quant method: {quant_method} (prefix: {amx_prefix})")
+
+    print(f"\nProcessing layers (max {args.max_tensors} tensors per shard)...")
+
+    accumulated_tensors = {}
+    shard_counter = {"shard": 1, "max_tensors": args.max_tensors}
+
+    for layer_idx, layer_folder in layer_folders:
+        layer_path = os.path.join(args.input_path, layer_folder)
+        print(f"Processing layer {layer_idx} ({layer_folder})...")
+
+        layer_tensors = process_layer(layer_path, amx_prefix, layer_idx)
+        print(f"  Loaded {len(layer_tensors)} tensors from this layer")
+
+        accumulated_tensors.update(layer_tensors)
+
+        if len(accumulated_tensors) >= args.max_tensors:
+            print(f"  Accumulator has {len(accumulated_tensors)} tensors, flushing to shard(s)...")
+            write_shards(accumulated_tensors, args.output, shard_counter, keep_remainder=True)
+
+        gc.collect()
+
+    if accumulated_tensors:
+        print(f"Flushing remaining {len(accumulated_tensors)} tensors to final shard(s)...")
+        write_shards(accumulated_tensors, args.output, shard_counter, keep_remainder=False)
+
+    if args.original_path:
+        print(f"\nCopying config files from {args.original_path}...")
+        copy_config_files(args.original_path, args.output)
+
+    total_shards = shard_counter["shard"] - 1
+    print(f"\nConversion completed! Created {total_shards} shard(s) in {args.output}")
+    return 0
+
+
+if __name__ == "__main__":
+    exit(main())