Ktransformers Doc

发表于 分类于 ，

基本信息

转载自：Ktransformers
防止丢失、学习记录备忘。
内容同步于 2026-11-10，对应官方文档最新版（v0.7.x）。

Introduction

KTransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations

🎯 Overview | 🚀 Inference | 🎓 SFT | 🔥 Citation | 🚀 Roadmap(2026Q2)

🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

🔥 Updates

  • Oct 15, 2026: DeepSeek-V4-Flash LoRA fine-tuning supports native MXFP4 routed experts. quickstart.
  • Oct 8, 2026: We release the author preprint of KTransformers-FineTune (KT-FT), presenting layout-aware and schedule-optimized CPU-GPU co-computation for MoE fine-tuning. Paper and citation · PDF.
  • Sep 13, 2026: Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts.
  • Aug 26, 2026: Added native support for GLM-5.3-flash, bringing 1M-token context and multimodal input to consumer GPUs. (Tutorial)
  • Aug 25, 2026: Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook, covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting.
  • Aug 17, 2026: LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. (v0.7.0 Release Notes)
  • Aug 16, 2026: DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. (Tutorial)
  • Aug 5, 2026: Introduced native block-FP8 LoRA fine-tuning (PR #2141)
  • July 23, 2026: Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. (PR #2094)
  • June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial)
  • June 17, 2026: GLM-5.2 Day0 Support! (Tutorial)
  • May 6, 2026: KTransformers at GOSIM Paris 2026 — “Agentic AI on Edge” track. We’ll present KT’s inference performance on consumer hardware.
  • May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial)
  • Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points.
  • Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial)
  • Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial)
  • Feb 12, 2026: GLM-5 Day0 Support! (Tutorial)
  • Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial)
  • Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference
  • Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial)
  • Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial)
  • Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial)
  • Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial)
  • Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial)
  • Oct 27, 2025: Support Ascend NPU. (Tutorial)
  • Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog)
  • Sept 11, 2025: Support Qwen3-Next. (Tutorial)
  • Sept 05, 2025: Support Kimi-K2-0905. (Tutorial)
  • July 26, 2025: Support SmallThinker and GLM4-MoE. (Tutorial)
  • July 11, 2025: Support Kimi-K2. (Tutorial)
  • June 30, 2025: Support 3-layer (GPU-CPU-Disk) prefix cache reuse.
  • May 14, 2025: Support Intel Arc GPU (Tutorial).
  • Apr 29, 2025: Support AMX-Int8、 AMX-BF16 and Qwen3MoE (Tutorial)
  • Apr 9, 2025: Experimental support for LLaMA 4 models (Tutorial).
  • Apr 2, 2025: Support Multi-concurrency. (Tutorial).
  • Mar 15, 2025: Support ROCm on AMD GPU (Tutorial).
  • Mar 5, 2025: Support unsloth 1.58/2.51 bits weights and IQ1_S/FP8 hybrid weights. Support 139K Longer Context for DeepSeek-V3 and R1 in 24GB VRAM.
  • Feb 25, 2025: Support FP8 GPU kernel for DeepSeek-V3 and R1; Longer Context.
  • Feb 15, 2025: Longer Context (from 4K to 8K for 24GB VRAM) & Slightly Faster Speed （+15%, up to 16 Tokens/s), update docs and online books.
  • Feb 10, 2025: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup. For detailed show case and reproduction tutorial, see here.
  • Aug 28, 2024: Decrease DeepseekV2’s required VRAM from 21G to 11G.
  • Aug 15, 2024: Update detailed tutorial for injection and multi-GPU.
  • Aug 14, 2024: Support llamfile as linear backend.
  • Aug 12, 2024: Support multiple GPU; Support new model: mixtral 8*7B and 8*22B; Support q2k, q3k, q5k dequant on gpu.
  • Aug 9, 2024: Support windows native.

📦 Capabilities

🚀 Inference - High-Performance kt-kernel Serving

CPU-optimized kernel operations for heterogeneous LLM inference.

image

Key Features:

  • AMX/AVX Acceleration: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
  • MoE Optimization: Efficient Mixture-of-Experts inference with NUMA-aware memory management
  • Quantization Support: CPU-side INT4/INT8 quantized weights, GPU-side GPTQ support
  • Easy Integration: Clean Python API for SGLang and other frameworks

Quick Start:

1
2
cd kt-kernel
pip install .

Use Cases:

  • CPU-GPU hybrid inference for large MoE models
  • Integration with SGLang for production serving
  • Heterogeneous expert placement (hot experts on GPU, cold experts on CPU)

Performance Examples:

Model Hardware Configuration Total Throughput Output Throughput
DeepSeek-R1-0528 (FP8) 8×L20 GPU + Xeon Gold 6454S 227.85 tokens/s 87.58 tokens/s (8-way concurrency)

👉 Full Documentation →

🎓 SFT - Fine-Tuning with LlamaFactory

Research paper: KTransformers-FineTune (KT-FT) · PDF.

KTransformers × LlamaFactory integration for ultra-large MoE model fine-tuning. The new Cookbook provides an easy-to-use path from hardware checks and installation to BF16/FP8/INT8 configuration, LoRA/full training, resource planning, and troubleshooting.

Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Follow the PyPI installation, training, resume and SGLang serving guide (中文). End-to-end validation uses Kimi K2.5; K2.6 follows the same model-architecture path.

LlamaFactory and KTransformers MoE fine-tuning architecture

Key Features:

  • Multi-Backend Support: CPU/GPU hybrid fine-tuning with INT8/INT4 quantization
  • Ultra-Large MoE Support: Fine-tune models like DeepSeek-V3/R1 on limited GPU memory
  • Faster than ZeRO-Offload: 6-12x training speedup in benchmarked MoE SFT workloads
  • Lower CPU Memory: About half the CPU memory of the previous KT SFT path in the benchmarked setup
  • LlamaFactory Integration: Seamless integration with the popular fine-tuning framework
Model GPU Memory Training Speed Hardware
DeepSeek-V3 ~80GB total 3.7 it/s 4x RTX 4090
DeepSeek-R1 ~80GB total 3.7 it/s 4x RTX 4090
Qwen3-30B-A3B ~24GB total 8+ it/s 1x RTX 4090

Quick Start:

1
2
3
4
5
6
7
8
cd /path/to/LLaMA-Factory
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

👉 Easy-to-Use Fine-Tuning Cookbook → 👉 Quick Start → 👉 Full Documentation →

🔥 Citation

If you use KTransformers in your research, please cite the relevant paper.

Inference paper

1
2
3
4
5
6
@inproceedings{10.1145/3731569.3764843,
  title = {KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models},
  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
  year = {2025}
}

Fine-tuning paper

KTransformers-FineTune (KT-FT) · Full BibTeX

1
2
3
4
5
6
7
8
@misc{li2026ktft,
  title = {{KTransformers-FineTune: Beyond Offload with Layout-Aware and Schedule-Optimized Heterogeneous MoE Fine-Tuning}},
  author = {Li, Peilin and Hao, Xingxing and Chen, Hongtao and Xie, Weiyu and Zheng, Yaowei and Wu, Bowen and Yang, Yujie and Shen, Huanming and Ou, Qingliang and Zhang, Boxin and Tang, Jingqi and Yuan, Ziwei and Dong, Jianwei and Kuang, Dongdong and Feng, Zhangchi and Dai, Jiaheng and Yang, Qianrui and Chen, Shaoyuan and Wang, Jiahao and Han, Yaochen and Zhu, Yuening and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  year = {2026},
  note = {Author preprint, version paper-ktft-v1.1},
  howpublished = {Author preprint, GitHub},
  url = {https://github.com/kvcache-ai/ktransformers/tree/paper-ktft-v1.1/papers/kt-ft}
}

👥 Contributors & Team

Developed and maintained by:

  • MADSys Lab @ Tsinghua University
  • Approaching.AI
  • 9#AISoft
  • Community contributors

We welcome contributions! Please feel free to submit issues and pull requests.

💬 Community & Support

  • GitHub Issues: Report bugs or request features
  • WeChat Group: See archive/WeChatGroup.png

📦 KT original Code

The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.

For the original documentation with full quick-start guides and examples, see:

  • archive/README.md (English)
  • archive/README_ZH.md (中文)

KT-Kernel

High-performance kernel operations for KTransformers, featuring CPU-optimized MoE inference with AMX, AVX, KML and blis (amd library) support.

  • Note
  • Features
  • Installation
    • Option 1: Install from PyPI (Recommended for Most Users)
    • Option 2: Install from Source (For Local Use or Custom Builds)
  • Verification
  • KT CLI Overview
  • Integration with SGLang
    • Installation Steps
    • Complete Example: Qwen3-30B-A3B
    • KT-Kernel Parameters
  • Direct Python API Usage
    • Advanced Options
    • Manual Configuration (Advanced)
  • Build Configuration
    • Manual Installation (Without install.sh)
  • Error Troubleshooting
    • CUDA Not Found
    • hwloc Not Found
  • Weight Quantization
  • Before Commit!

Note

Current Support Status:

  • ✅ Native Precision with AVX512/AMX: Supported with AVX512 CPUs in FP8, BF16 and RAWINT4 format - Guide
  • ✅ Intel CPUs with AMX: Fully supported (using weights converted to INT4/INT8 format)
  • ✅ Universal CPU (llamafile backend): Supported (using GGUF-format weights)
  • ✅ AMD CPUs with BLIS: Supported (for int8 prefill & decode) - Guide

KT-CLI

We are developing a simpler way to use KTransformers. Check out the KT-CLI Guide for more details.

Features

  • CPU-Optimized MoE Kernels: High-throughput MoE expert kernels optimized for instruction sets.
  • AVX512 Native Precision Backend: FP8 / BF16 / INT4 native MoE backend for AVX512-capable servers.
  • AMX INT4/INT8 Backend: INT4 / INT8 quantized expert inference backend for AMX-capable servers.
  • Llamafile CPU Backend: AVX2/AVX512-based MoE backend built on Llamafile for universal CPU deployment.
  • NUMA-Aware Execution: Thread pool and memory layout designed for multi-socket / multi-NUMA machines.

Installation

Install the latest version with a single command:

1
pip install kt-kernel

Note: Check the latest version on PyPI

Features:

  • ✅ Automatic CPU detection: Detects your CPU and loads the optimal kernel variant
  • ✅ CPU multi-variant support: Includes AMX, AVX512 (Base/VNNI/VBMI/BF16), and AVX2 variants
  • ✅ CUDA support included: GPU acceleration for NVIDIA GPUs (SM 80, 86, 89, 90)
  • ✅ No compilation needed: Pre-built wheels for Python 3.10, 3.11, 3.12
  • ✅ Static CUDA runtime: No CUDA toolkit installation required
  • ✅ Works on CPU-only systems: CUDA features automatically disabled when GPU not available

Requirements:

  • Python 3.10, 3.11, or 3.12
  • Linux x86-64 (manylinux_2_17 compatible)
  • CPU with AVX2 support (Intel Haswell 2013+, AMD Zen+)
  • Optional: NVIDIA GPU with compute capability 8.0+ for CUDA features
CUDA Installation (GPU Acceleration)

For NVIDIA GPU-accelerated inference:

1
pip install kt-kernel

Features:

  • ✅ Multi-architecture support: Single wheel supports SM 80/86/89/90 (Ampere, Ada, Hopper)
  • ✅ Static CUDA runtime: No CUDA toolkit installation required
  • ✅ Broad compatibility: Works with CUDA 11.8+ and 12.x drivers
  • ✅ PyTorch compatible: Works with any PyTorch CUDA variant (cu118, cu121, cu124)

Requirements:

  • Python 3.10, 3.11, or 3.12
  • Linux x86-64 (manylinux_2_17 compatible)
  • NVIDIA GPU with compute capability 8.0+ (Ampere or newer)
    • ✅ Supported: A100, RTX 3000/4000 series, H100
    • ❌ Not supported: V100, P100, GTX 1000/2000 series (too old)
  • NVIDIA driver with CUDA 11.8+ or 12.x support (no CUDA toolkit needed)

GPU Compatibility Matrix:

GPU Architecture Compute Capability Supported Example GPUs
Hopper 9.0 ✅ H100, H200
Ada Lovelace 8.9 ✅ RTX 4090, 4080, 4070
Ampere 8.6 ✅ RTX 3090, 3080, 3070, 3060
Ampere 8.0 ✅ A100, A30
Turing 7.5 ❌ RTX 2080, T4
Volta 7.0 ❌ V100

CUDA Driver Compatibility (for GPU features):

  • CUDA 11.8, 11.9, 12.0-12.6+: Full support
  • CUDA 11.0-11.7: Not supported (upgrade driver or use CPU-only)

CPU Variants Included:

The wheel includes 6 optimized variants that are automatically selected at runtime based on your CPU:

Variant CPU Support Performance Auto-Selected When
AMX Intel Sapphire Rapids+ (2023+) ⚡⚡⚡ Best AMX instructions detected
AVX512+BF16 Ice Lake server, Zen 4+ (2021+) ⚡⚡⚡ Excellent AVX512 + BF16 detected
AVX512+VBMI Ice Lake client (2019+) ⚡⚡ Great AVX512 + VBMI detected
AVX512+VNNI Cascade Lake+ (2019+) ⚡⚡ Great AVX512 + VNNI detected
AVX512 Base Skylake-X+ (2017+) ⚡⚡ Good AVX512 base detected
AVX2 Haswell+ (2013+), AMD Zen+ ⚡ Good Fallback for maximum compatibility

Verify installation:

1
2
3
4
5
6
7
8
9
10
11
12
13
import kt_kernel

# Check which CPU variant was loaded
print(f"CPU variant: {kt_kernel.__cpu_variant__}")
print(f"Version: {kt_kernel.__version__}")

# Check CUDA support
from kt_kernel import kt_kernel_ext
cpu_infer = kt_kernel_ext.CPUInfer(4)
has_cuda = hasattr(cpu_infer, 'submit_with_cuda_stream')
print(f"CUDA support: {has_cuda}")

print("✓ kt-kernel installed successfully!")

Environment Variables:

1
2
3
4
5
6
# Override automatic CPU detection (for testing or debugging)
export KT_KERNEL_CPU_VARIANT=avx2  # Force specific variant

# Enable debug output to see detection process
export KT_KERNEL_DEBUG=1
python -c "import kt_kernel"

Option 2: Install from Source (For Local Use or Custom Builds)

Build from source for local installation or when you need AMD (BLIS), ARM (KML), or custom CUDA versions.

Prerequisites

First, initialize git submodules and create a conda environment:

1
2
3
git submodule update --init --recursive
conda create -n kt-kernel python=3.11 -y
conda activate kt-kernel

Simply run the install script - it will auto-detect your CPU and optimize for best performance:

1
./install.sh

What happens automatically:

  • Auto-detects CPU capabilities (AMX, AVX512_VNNI, AVX512_BF16)
  • Installs system dependencies (cmake, libhwloc-dev, pkg-config)
  • Builds optimized binary for your CPU only (using -march=native)
  • Software fallbacks: Automatically enabled for CPUs without VNNI/BF16

Optional: Two-step installation

1
2
./install.sh deps   # Install dependencies only
./install.sh build  # Build and install kt-kernel

CPU Requirements by Backend:

Backend Minimum CPU Requirement Example CPUs Notes
LLAMAFILE AVX2 Intel Haswell (2013+), AMD Zen+ Universal compatibility
RAWINT4 AVX512F + AVX512BW Intel Skylake-X (2017+), Ice Lake, Cascade Lake Software fallbacks for VNNI/BF16
AMXINT4/INT8 AMX Intel Sapphire Rapids (2023+) Best performance, requires AMX hardware
FP8 AVX512F + AVX512BW + AVX512_BF16 + AVX512_VBMI Intel Cooper Lake (2020+), Sapphire Rapids (2023+); AMD Zen 4+ (e.g., EPYC 9355) Native Precision (e.g., DeepSeek V3.2, MiniMax M2.1)
BF16 AVX512F + AVX512BW + AVX512_BF16 Intel Cooper Lake (2020+), Sapphire Rapids (2023+); AMD Zen 4+ (e.g., EPYC 9355) Native Precision (e.g., Qwen3-235B-A22B, GLM-4.7)

Software Fallback Support (AVX512 backends):

  • ✅ VNNI fallback: Uses AVX512BW instructions
  • ✅ BF16 fallback: Uses AVX512F instructions
  • ✅ Older AVX512 CPUs (Skylake-X, Cascade Lake) can run RAWINT4 with fallbacks

⚠️ Portability Note: The default build is optimized for your specific CPU and may not work on different/older CPUs. For portable builds or binary distribution, see Manual Configuration below.

⚠️ AMD BLIS backend users: See installation guide for AMD-specific setup.

Verification

After installation, verify that the CLI is working:

1
kt version

Expected output:

1
2
3
4
5
6
7
KTransformers CLI v0.x.x

  Python:        3.11.x
  Platform:      Linux 5.15.0-xxx-generic
  CUDA:          12.x
  kt-kernel:     0.x.x (amx)
  sglang:        0.x.x

You can also verify the Python module directly:

1
python -c "from kt_kernel import KTMoEWrapper; print('✓ kt-kernel installed successfully')"

KT CLI Overview

The kt command-line tool provides a unified interface for running and managing KTransformers models:

Command Description
kt run Start model inference server with auto-optimized parameters
kt chat Interactive chat with a running model server
kt model Manage models and storage paths
kt doctor Diagnose environment issues and check system compatibility
kt config Manage CLI configuration
kt version Show version information

Quick Start Example:

1
2
3
4
5
6
7
8
# Start a model server (auto-detects hardware and applies optimal settings)
kt run m2

# In another terminal, chat with the model
kt chat

# Check system compatibility
kt doctor

Run kt --help for more options, or kt <command> --help for command-specific help.

Integration with SGLang

KT-Kernel can be used standalone via Direct Python API or integrated with SGLang for production deployment. This section describes SGLang integration to enable CPU-GPU heterogeneous inference, where “hot” experts run on GPU and “cold” experts run on CPU for optimal resource utilization.

Installation Steps

1. Install SGLang

Install the kvcache-ai fork of SGLang (required for kt-kernel support):

1
2
3
4
5
6
7
8
9
10
# Option A: One-click install (from ktransformers root, installs sglang + kt-kernel)
./install.sh

# Option B: pip install
pip install kt-kernel sglang-kt

# Option C: From source (editable mode)
git clone --recursive https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install -e "third_party/sglang/python[all]"

Important: Use sglang-kt (kvcache-ai fork), not the official sglang package. If you have the official version installed, uninstall it first: pip uninstall sglang -y

2. Prepare Weights

You need both GPU weights and CPU-side expert weights for heterogeneous inference. The exact format depends on the backend:

GPU Weights (for all backends):
Use the model weights required by SGLang for GPU inference (for example, the original or already-quantized model directory from Hugging Face).

CPU Weights (AMX backend: AMXINT4 / AMXINT8): Quantize weights to AMX-optimized INT4/INT8 format using the provided script:

1
2
3
4
5
python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \
  --output /path/to/cpu-weights \
  --quant-method int8  # or int4 or moe_int8 (for amd now)
  • --input-path: Path to GPU-side original weights
  • --input-type: Depends on your GPU weights type (fp8, fp16, or bf16)

In SGLang integration, --kt-weight-path should point to this converted CPU weights directory.

Supported input formats: FP8, FP16, BF16 → INT4/INT8.

CPU Weights (LLAMAFILE backend: LLAMAFILE): LLAMAFILE uses pre-quantized GGUF weights on the CPU side directly, without running convert_cpu_weights.py. You need to:

  • Download a GGUF model directly from the web (e.g., GGUF repos on Hugging Face / Modelscope);
  • In SGLang integration, use that GGUF directory as --kt-weight-path. KT-Kernel supports multiple GGUF quantization formats such as Q4_KM, Q4_K, Q5_K, etc. Choose based on your latency and accuracy requirements.
3. Launch SGLang Server

Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference:

KT-Kernel Parameters to Add:

  • --kt-method: Backend method (AMXINT4, AMXINT8, or LLAMAFILE)
  • --kt-weight-path: Path to the converted CPU weights
  • --kt-cpuinfer: Number of CPU inference threads (set to physical cores)
  • --kt-threadpool-count: Number of thread pools (set to NUMA node count)
  • --kt-num-gpu-experts: Number of experts to keep on GPU
  • --kt-max-deferred-experts-per-token: Deferred experts for pipelined execution

Example:

1
2
3
4
5
6
7
8
python -m sglang.launch_server \
  [your normal SGLang parameters...] \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/cpu-weights \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

See KT-Kernel Parameters section below for detailed parameter tuning guidelines.

Complete Example: Qwen3-30B-A3B

This example demonstrates the full workflow from downloading weights to launching the server, showing Native backend, AMX backend and LLAMAFILE backend options.

Hardware Configuration:

  • GPU: NVIDIA RTX 4090 24GB
  • CPU: 2x Intel Xeon Gold 6454S (64 physical cores total, 128 threads, 2 NUMA nodes)
  • Model: Qwen3-30B-A3B

How to verify your system configuration:

1
2
3
4
5
6
7
8
# Check CPU configuration
lscpu | grep -E "^CPU(s)|Thread(s) per core|Socket(s)|NUMA node(s)"
# Expected output example:
CPU(s):                                  128
Thread(s) per core:                      2
Socket(s):                               2
NUMA node(s):                            2
# → Physical cores = CPU(s) / Thread(s) per core = 128 / 2 = 64

Parameter Rationale:

  • --kt-cpuinfer 64: Set to physical cores (64), not hyperthreads (128)
  • --kt-threadpool-count 2: 2 NUMA nodes detected (dual-socket system)
  • --kt-num-gpu-experts 32: With 24GB GPU memory, we can fit ~32 experts on GPU for this model (varies by model architecture and actual memory usage)
  • --kt-max-deferred-experts-per-token 2: Enable pipelined execution; allows CPU to process next batch while GPU completes current batch
  • --kt-gpu-prefill-token-threshold 2048: Use layerwise prefill strategy when token count exceeds 2048 (for native backends only)
Option A: Native Backend (BF16)

For AVX512 CPUs with BF16 support.

Step 1: Download model weights

1
2
3
4
# Install huggingface-cli if not already installed
pip install huggingface-hub
# Download model from Hugging Face
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Launch SGLang server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
python -m sglang.launch_server \
    --host 0.0.0.0 \
    --port 30000 \
    --model /mnt/data/models/Qwen3-30B-A3B \
    --kt-weight-path /mnt/data/models/Qwen3-30B-A3B \
    --kt-cpuinfer 64 \
    --kt-threadpool-count 2 \
    --kt-num-gpu-experts 32 \
    --kt-method BF16 \
    --attention-backend flashinfer \
    --trust-remote-code \
    --mem-fraction-static 0.80 \
    --chunked-prefill-size 16384 \
    --max-running-requests 4 \
    --served-model-name Qwen3 \
    --enable-mixed-chunk \
    --tensor-parallel-size 1 \
    --enable-p2p-check \
    --disable-shared-experts-fusion \
    --kt-gpu-prefill-token-threshold 4096 \
    --kt-enable-dynamic-expert-update
Option B: AMX Backend (AMXINT8)

For Intel CPUs with AMX instruction set support.

Step 1: Download model weights

1
2
3
4
5
# Install huggingface-cli if not already installed
pip install huggingface-hub

# Download model from Hugging Face
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Convert to CPU weights (AMXINT8)

1
2
3
4
5
python scripts/convert_cpu_weights.py \
  --input-path /mnt/data/models/Qwen3-30B-A3B \
  --input-type bf16 \
  --output /mnt/data/models/Qwen3-30B-A3B-INT8 \
  --quant-method int8

Step 3: Launch SGLang server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model /mnt/data/models/Qwen3-30B-A3B \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name Qwen3-30B-A3B \
  --enable-mixed-chunk \
  --kt-method AMXINT8 \
  --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-INT8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2
Option C: LLAMAFILE Backend (GGUF)

For universal CPUs (no AMX required), using pre-quantized GGUF weights directly.

Step 1: Download GPU weights (original model)

1
2
3
pip install huggingface-hub

huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B

Step 2: Download CPU weights (GGUF format)

1
2
huggingface-cli download Qwen/Qwen3-30B-A3B-GGUF Qwen3-30B-A3B-Q4_K_M.gguf \
  --local-dir /mnt/data/models/Qwen3-30B-A3B-Q4_K_M

Step 3: Launch SGLang server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model /mnt/data/models/Qwen3-30B-A3B \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name Qwen3-30B-A3B \
  --enable-mixed-chunk \
  --kt-method LLAMAFILE \
  --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-Q4_K_M \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

KT-Kernel Parameters

Parameter Description Example Value
--kt-method CPU inference backend method AMXINT4, AMXINT8, RAWINT4, FP8, FP8_PERCHANNEL, BF16 or LLAMAFILE
--kt-weight-path Path to quantized CPU weights /path/to/cpu-weights
--kt-cpuinfer Number of CPU inference threads 64 (adjust based on CPU cores)
--kt-threadpool-count Number of thread pools for parallel execution 2 (typically 1-4)
--kt-num-gpu-experts Number of experts to keep on GPU 32 (remaining experts go to CPU)
--kt-max-deferred-experts-per-token Number of experts per token to defer for pipelined execution 2 (0 to disable, 1-4 recommended)
--kt-gpu-prefill-token-threshold Token count threshold for prefill strategy (native backend only) ~`1024-4096`
--kt-enable-dynamic-expert-update Enable dynamic expert placement updates during prefill based on actual routing statistics (flag, no value needed)
--kt-expert-placement-strategy Strategy for initial GPU expert placement uniform, frequency, front-loading, or random

Parameter Guidelines:

  • kt-method: Choose based on your CPU and weight format:

    • AMXINT4: Best performance on AMX CPUs with INT4 quantized weights (May cause huge accuracy drop for some models, e.g., Qwen3-30B-A3B)
    • AMXINT8: Higher accuracy with INT8 quantized weights on AMX CPUs
    • RAWINT4: Native INT4 weights shared by CPU and GPU (currently supports Kimi-K2-Thinking model). See Kimi-K2-Thinking Native Tutorial for details.
    • FP8, FP8_PERCHANNEL: FP8 weights shared by CPU and GPU
    • BF16: BF16 weights shared by CPU and GPU
    • LLAMAFILE: GGUF-based backend

  • kt-cpuinfer: Set to the number of physical CPU cores (not hyperthreads).

    • Check physical cores: lscpu | grep -E "^CPU(s)|Thread(s) per core"
    • Physical cores = CPU(s) / Thread(s) per core
    • Example: If CPU(s)=128 and Thread(s) per core=2, then physical cores = 64
    • Important: Do NOT set to hyperthread count - this will degrade performance

  • kt-threadpool-count: Set to the number of NUMA nodes.

    • Check NUMA count: lscpu | grep "NUMA node(s)"
    • Or use: numactl --hardware | grep "available"
    • Note: NUMA node count is NOT necessarily the number of physical CPUs
      • It represents memory domains, which may be divided within a single CPU or across multiple CPUs
      • Use the NUMA node count from lscpu, regardless of physical CPU count
    • Typical values: 1-2 for single-socket, 2-4 for dual-socket systems
    • This enables better memory bandwidth utilization across NUMA domains

  • kt-num-gpu-experts: Determine based on GPU memory and profiling:

    • More GPU experts = lower latency but higher GPU memory usage (May cause OOM)

  • kt-max-deferred-experts-per-token: Enables pipelined execution:

    • 0: Synchronous execution (simpler, higher latency)
    • 1-4: Deferred execution (recommended range; good latency/quality balance, requires tuning)
    • 5-7: Highest latency reduction but may introduce noticeable accuracy loss; use with care

  • kt-gpu-prefill-token-threshold (FP8 and RAWINT4 only): Controls prefill strategy for native FP8 and INT4 inference:

    • ≤ threshold: Uses hybrid CPU+GPU prefill. No extra VRAM needed, but performance degrades slowly as token count increases.
    • > threshold: Uses layerwise GPU prefill. Performance scales better with longer sequences, but requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking and ~3.6GB for MiniMax-M2.1).
    • Only applicable when --kt-method RAWINT4 or --kt-method FP8 is used.

  • kt-enable-dynamic-expert-update: Enables dynamic expert placement updates during inference.

    • During layerwise prefill, the system collects actual routing statistics and redistributes GPU experts accordingly.
    • Requires --kt-gpu-prefill-token-threshold to be set, and prefill length must be ≥ the threshold value.
    • Particularly effective at lower GPU expert ratios (10%-70%), where it can significantly outperform static strategies.
    • See Expert Scheduling Tutorial for benchmarks and details.

  • kt-expert-placement-strategy: Determines which experts are placed on GPU at server startup.

    • uniform: Distributes GPU experts evenly across all MoE layers. Default option, no prior statistics needed.
    • frequency: Places the most frequently activated experts on GPU. Best performance when activation statistics are available; requires --init-expert-location pointing to a .pt statistics file.
    • front-loading: Fills GPU experts from the first MoE layer onwards.
    • random: Randomly selects experts with a fixed seed (42).
    • See Expert Scheduling Tutorial for strategy comparison.

Direct Python API Usage

For standalone usage without SGLang, you can use KT-Kernel directly via Python API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from kt_kernel import KTMoEWrapper

# Initialize the MoE wrapper
wrapper = KTMoEWrapper(
    layer_idx=0,
    num_experts=8,
    num_experts_per_tok=2,
    hidden_size=4096,
    moe_intermediate_size=14336,
    num_gpu_experts=2,
    cpuinfer_threads=32,
    threadpool_count=2,
    weight_path="/path/to/weights",
    chunked_prefill_size=512,
    method="AMXINT4"  # Options: "AMXINT4", "AMXINT8", "LLAMAFILE"
)

# Load weights (from disk - pre-quantized)
wrapper.load_weights(physical_to_logical_map)

# Or load weights from tensors (online quantization)
wrapper.load_weights_from_tensors(gate_proj, up_proj, down_proj, physical_to_logical_map)

# Run inference
output = wrapper.forward(hidden_states, topk_ids, topk_weights, cuda_stream)

# Or use async API for better performance
wrapper.submit_forward(hidden_states, topk_ids, topk_weights, cuda_stream)
# ... do other work ...
output = wrapper.sync_forward(hidden_states, cuda_stream)

Advanced Options

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Initialize with additional options
wrapper = KTMoEWrapper(
    layer_idx=0,
    num_experts=8,
    num_experts_per_tok=2,
    hidden_size=4096,
    moe_intermediate_size=14336,
    num_gpu_experts=2,
    cpuinfer_threads=32,
    threadpool_count=2,
    weight_path="/path/to/weights",
    chunked_prefill_size=512,
    method="AMXINT4",
    cpu_save=False,  # Keep weights in CPU memory after loading
    max_deferred_experts_per_token=0  # Number of experts to defer (for pipelined execution)
)

# Pre-allocate buffers for specific batch sizes (improves performance)
KTMoEWrapper.set_capture_batch_sizes([1, 2, 4, 8, 16])

# Query captured batch sizes
batch_sizes = KTMoEWrapper.get_capture_batch_sizes()

# Clear buffer cache to free memory
KTMoEWrapper.clear_buffer_cache()

Manual Configuration (Advanced)

For portable builds, binary distribution, or cross-machine deployment, you need to manually specify target instruction sets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# General distribution (works on any AVX512 CPU from 2017+)
export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

# Maximum compatibility (works on any CPU from 2013+)
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

# Modern CPUs only (Ice Lake+, Zen 4+)
export CPUINFER_CPU_INSTRUCT=FANCY
export CPUINFER_ENABLE_AMX=OFF
./install.sh build --manual

Optional: Override VNNI/BF16 detection

1
2
3
4
# Force enable/disable VNNI and BF16 (for testing fallbacks)
export CPUINFER_ENABLE_AVX512_VNNI=OFF
export CPUINFER_ENABLE_AVX512_BF16=OFF
./install.sh

Optional: oneDNN INT8 BRGEMM on non-AMX AVX512 CPUs

1
2
3
4
5
6
7
8
9
# Requires an installed oneDNN >= 3.9 package:
export CPUINFER_ENABLE_ONEDNN_VNNI=ON

# Or build against a oneDNN source tree:
export CPUINFER_ONEDNN_SOURCE_DIR=/path/to/oneDNN
./install.sh

# The compiled wheel defaults to oneDNN; native remains available for A/B tests.
export KT_INT8_VNNI_BACKEND=auto  # auto | onednn | native

See ./install.sh --help for all available options.

Build Configuration

Manual Installation (Without install.sh)

If you prefer manual installation without the install.sh script:

1. Install System Dependencies

Prerequisites:

  • cmake (recommended: conda install -y cmake)
  • libhwloc-dev and pkg-config
2. Set Build Configuration

Core Options:

Variable Options Description
CPUINFER_CPU_INSTRUCT NATIVE, AVX512, AVX2, FANCY CPU instruction set to use
CPUINFER_ENABLE_AMX ON, OFF Enable Intel AMX support
CPUINFER_ENABLE_ONEDNN_VNNI ON, OFF Enable oneDNN INT8 BRGEMM on non-AMX AVX512 CPUs
CPUINFER_ONEDNN_SOURCE_DIR Path Optional oneDNN >= 3.9 source tree
CPUINFER_BUILD_TYPE Release, Debug, RelWithDebInfo Build type (default: Release)
CPUINFER_PARALLEL Number Parallel build jobs (default: auto-detect)
CPUINFER_VERBOSE 0, 1 Verbose build output (default: 0)

Instruction Set Details:

Option Target CPUs Use Case
NATIVE Your specific CPU only Local builds (best performance, default)
AVX512 Skylake-X, Ice Lake, Cascade Lake, Zen 4+ General distribution
AVX2 Haswell (2013) and newer Maximum compatibility
FANCY Ice Lake+, Zen 4+ Modern CPUs with full AVX512 extensions

Example Configurations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Local use - maximum performance (default behavior)
export CPUINFER_CPU_INSTRUCT=NATIVE
export CPUINFER_ENABLE_AMX=ON  # or OFF

# Distribution build - works on any AVX512 CPU
export CPUINFER_CPU_INSTRUCT=AVX512
export CPUINFER_ENABLE_AMX=OFF

# Maximum compatibility - works on CPUs since 2013
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF

# Debug build
export CPUINFER_BUILD_TYPE=Debug
export CPUINFER_VERBOSE=1
3. Build and Install
1
2
3
4
5
# Editable installation (for development)
pip install -e .

# Standard installation
pip install .

Error Troubleshooting

CUDA Not Found

1
2
3
-- Looking for a CUDA compiler - NOTFOUND
 CMake Error at CMakeLists.txt:389 (message):
   KTRANSFORMERS_USE_CUDA=ON but CUDA compiler not found

Make sure you have the CUDA toolkit installed and nvcc is in your system PATH.

Try export CMAKE_ARGS="-D CMAKE_CUDA_COMPILER=$(which nvcc)" and reinstall again.

hwloc Not Found

Run sudo apt install libhwloc-dev if on a Debian-based system or build from source: https://www.open-mpi.org/projects/hwloc/.

1
2
3
4
5
6
wget https://download.open-mpi.org/release/hwloc/v2.12/hwloc-2.12.2.tar.gz
tar -xzf hwloc-2.12.2.tar.gz
cd hwloc-2.12.2
./configure
make
sudo make install

Weight Quantization

For AMX backends (AMXINT4 / AMXINT8), CPU-side experts must be converted to AMX-friendly INT4/INT8 format using the provided script:

1
2
3
4
5
python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \
  --output /path/to/output \
  --quant-method int4

Supported formats: FP8, FP16, BF16 → INT4/INT8

For LLAMAFILE backend (LLAMAFILE), CPU-side experts are loaded directly from GGUF weights. You do not need to run the AMX conversion script; instead, download a GGUF model from the web (e.g., a GGUF repo on Hugging Face) and point weight_path / SGLang --kt-weight-path (or --model when appropriate) to that GGUF directory. KT-Kernel supports multiple GGUF quantization types such as Q4_KM, Q4_K, Q5_K, etc.

For detailed documentation, advanced options, and low-memory mode, see scripts/README.md.

Before Commit!

Commit messages should follow the Conventional Commits specification: https://www.conventionalcommits.org/

Please format your code before committing:

1
2
3
cmake -B build
cd build
make format

You may need a newer clang-format (at least version 18). In a conda environment:

1
2
conda install -c conda-forge clang-format=18
rm -rf build

It’s also recommended to install black for Python code formatting:

1
conda install black

KTransformers Fine-Tuning × LLaMA-Factory Integration – User Guide

MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team

Introduction

From DeepSeek-V3/R1 to Qwen3-MoE and Kimi-K2, each wave of open-sourced large models brings leaps in performance and scale. However, many researchers and developers are constrained by expensive GPUs and models with tens or even hundreds of billions of parameters, making it hard to fine-tune very large models under limited resources. To bridge this gap, we propose a practical approach: combining KTransformers with LLaMA-Factory. With just 2–4 RTX 4090s and a high-memory CPU, you can fine-tune ultra-large MoE models like DeepSeek-671B.

Our goal is to give resource-constrained researchers a local path to explore fine-tuning ultra-large models, and also a fast way to customize smaller models (e.g., 14B/30B) for specific scenarios. We validate the setup using stylized dialogue, Westernized translation tone, and medical Q&A as representative tasks, showing that personalized adaptation can be achieved within hours.

As shown below, LLaMA-Factory is the unified orchestration/configuration layer for the whole fine-tuning workflow—handling data, training scheduling, LoRA injection, and inference interfaces. KTransformers acts as a pluggable high-performance backend that takes over core operators like Attention/MoE under the same training configs, enabling efficient GPU+CPU heterogeneous cooperation.

image-20251011010558909

Within LLaMA-Factory, we compared LoRA fine-tuning with HuggingFace, Unsloth, and KTransformers backends. KTransformers is the only workable 4090-class solution for ultra-large MoE models (e.g., 671B) and also delivers higher throughput and lower GPU memory on smaller MoE models (e.g., DeepSeek-14B).

Under LoRA (BF16) + NekoQA-10K stylized dialogue HuggingFace Backend Unsloth Backend KTransformers Backend
[14B-DeepSeekV2-Lite] LoRA fine-tuning throughput 303.58 token/s 455.37 token/s 530.38 token/s
[14B-DeepSeekV2-Lite] GPU memory 32.12 GB 9.64 GB 6.08 GB
[671B-DeepSeekV3] LoRA fine-tuning throughput Too Huge to run NOT SUPPORT 40.35 token/s
[671B-DeepSeekV3] GPU memory (sum across GPUs) theoretical 1400 GB † NOT SUPPORT 70 GB †

† 1400 GB is a theoretical FP16 full-parameter resident footprint (not runnable). 70 GB is the measured peak with KT strategy (Attention on GPU + layered MoE offload).

按照模型划分的对比图_02

Fine-Tuning Results (Examples)

Stylized Dialogue (CatGirl tone)

Dataset: NekoQA-10K. Goal: improve style consistency and recognizability.

The figure compares responses from the base vs. fine-tuned models. The fine-tuned model maintains the target tone and address terms more consistently (red boxes), validating the effectiveness of style-transfer fine-tuning.

image-20251016175046882

Benchmarks

We use:

(1) Translational-Style-ChatLLM, which asks for an exaggerated, Westernized translation tone—clear, stylized customization.

(2) AfriMed-QA (ACL 2025), a medical dataset for African contexts with strong domain specificity, including multiple-choice and short-answer sub-tasks—well-suited for vertical fine-tuning evaluation.

The tables show metrics before vs. after LoRA fine-tuning. We observe large improvements across metrics, verifying fine-tuning effectiveness:

Translational-Style dataset BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L
V2-Lite (no LoRA) 20.66 8.33 4.54 2.89 22.71 4.52 19.19
KT-LoRA fine-tuned V2-Lite 35.41 22.44 15.42 11.18 42.03 18.38 33.10
V3 base (no LoRA) 8.49 3.34 1.62 0.96 15.91 2.55 10.07
KT-LoRA fine-tuned V3 37.02 23.70 16.21 11.49 43.43 18.96 34.54
AfriMed-QA (short answer) BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L
V2-Lite (no LoRA) 13.58 11.12 9.10 7.23 22.48 7.81 11.73
KT-LoRA fine-tuned V2-Lite 35.90 27.63 22.99 19.15 35.25 17.50 28.44
V3 base (no LoRA) 12.75 10.27 8.05 5.99 20.33 5.65 10.11
KT-LoRA fine-tuned V3 42.42 34.12 28.95 24.54 41.97 22.37 33.28
AfriMed-QA (multiple choice) Accuracy
V2-Lite (no LoRA) 0.0645
KT-LoRA fine-tuned V2-Lite 0.4812
V3 base (no LoRA) 0.5833
KT-LoRA fine-tuned V3 0.7930

Even for ultra-large MoE models, KTransformers-backed fine-tuning achieves strong task performance quickly.

Quick to Start

This section shows how to install and use LLaMA-Factory + KTransformers for fine-tuning and inference:

  • Environment setup
  • Fine-tune ultra-large MoE models with KTransformers backend
  • Load the fine-tuned model (base + LoRA adapter) for chat/inference
  • Batch inference and metric evaluation

Environment Setup

According to the following example, install both the KTransformers and LLaMA-Factory environments simultaneously. This time, to simplify the installation process of KTransformers, use the PyPI packages to avoid local compilation. The detailed installation steps are as follows: (Note: Make sure your local Python version, Torch version, and CUDA version are compatible with the installed packages.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 1. Create a conda environment
conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13]
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime

# 2. Install the LLaMA-Factory environment
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .

# 3. Install the KTransformers SFT packages
pip install "ktransformers[sft]"

# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases
pip install flash-attn --no-build-isolation
# abi=True/False can find from below
# import torch
# print(torch._C._GLIBCXX_USE_CXX11_ABI)

# 5. (Optional) If you want to use flash_infer (otherwise it defaults to triton)
git clone https://github.com/kvcache-ai/custom_flashinfer.git
pip install custom_flashinfer/

Usage tip: In LLaMA-Factory YAML, set use_kt: true and pick a kt_optimize_rule file to have KTransformers handle the core compute. The features below show typical configs.

Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models

Run the command: USE_KT=1 ACCELERATE_USE_KT=true accelerate launch --config_file examples/ktransformers/accelerate/fsdp2_kt_bf16.yaml -m llamafactory.cli train examples/ktransformers/train_lora/deepseek_v3_lora_sft_kt.yaml.

Note: You must provide a BF16 model. DeepSeek-V3-671B is released in FP8 by default; convert with DeepSeek-V3/inference/fp8_cast_bf16.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
### model

model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
trust_remote_code: true

### method

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset

dataset: identity
template: deepseek
cutoff_len: 2048
max_samples: 100000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output

output_dir: saves/Kllama_deepseekV3
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### ktransformers

use_kt: true # use KTransformers as LoRA sft backend
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

kt_optimize_rule controls placement strategy. See also ktransformers/optimize_rules. Naming hints (* = wildcard):

Pattern Meaning
DeepSeek-V2-Lite-Chat-* / DeepSeek-V3-Chat-* Target model variants
-sft- Strategy for fine-tuning; others are for inference
-amx- Use AMX on CPU; otherwise use llamafile
-multi-gpu-X Model parallel on X GPUs (X omitted → default 2 GPUs)

Example: DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml = V3-Chat fine-tuning with AMX and 2-GPU model parallel.

We recommend AMX acceleration where available (lscpu | grep amx). AMX supports BF16/INT8. Example:

1
2
3
4
5
6
7
8
9
10
11
- match:
    name: "^model\.layers\..*\.mlp\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert parallelism
    kwargs:
      prefill_device: "cpu"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op: "KSFTExpertsCPU"
      out_device: "cuda"
      backend: "AMXInt8" # or "AMXBF16" or "llamafile" (default)

Outputs go to output_dir in safetensors format plus adapter metadata for later loading.

image-20251016171537997

Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter)

Run the command: llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml.

Use the safetensors adapter trained with KT for inference.

1
2
3
4
5
6
7
8
9
10
model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
adapter_name_or_path: saves/Kllama_deepseekV3
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

We also support GGUF adapters: for safetensors, set the directory; for GGUF, set the file path in adapter_name_or_path.

During loading, LLaMA-Factory maps layer names to KT’s naming. You’ll see logs like Loaded adapter weight: XXX -> XXX:

image-20251016171526210

Core Feature 3: Batch inference + metrics (base + LoRA adapter)

Run the command: API_PORT=8000 llamafactory-cli api examples/inference/qwen3_lora_sft.yaml. Invoke the KT fine-tuned adapter to provide the API; the usage logic of other APIs is consistent with the native LLaMA-Factory approach.

1
2
3
4
5
6
7
8
9
10
model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
adapter_name_or_path: saves/Kllama_deepseekV3
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
cpu_infer: 32
chunk_size: 8192

KT Fine-Tuning Speed (User-Side View)

End-to-End Performance

Definitions

  • step_time: wall-clock time for a full optimization step (tensor movement + Attention + MoE + other compute).
  • tokens_per_step = GAS × qlen; token/s = tokens_per_step / step_time.

Settings: GAS=16, qlen=512 (→ tokens_per_step = 8192); LoRA (r=8, alpha=32, dropout=0.1); AMX enabled; GPU: RTX 4090, CPU: Intel Xeon Platinum 8488C.

Measured

  • DeepSeek-V3-671B: step_time = 203 s → token/s ≈ 8192 / 203 ≈ 40.35
  • DeepSeek-V2-Lite-14B: step_time = 36 s → token/s ≈ 8192 / 36 ≈ 227.6

GPU/CPU Memory Footprint

  • DeepSeek-V3 (671B; 61 layers with 58 MoE): ~70 GB total GPU VRAM (multi-GPU), ~1.2–1.3 TB CPU RAM.
  • DeepSeek-V2-Lite (14B; 27 layers with 26 MoE): ~5.5 GB GPU VRAM, ~30 GB CPU RAM.

Conclusion

By integrating KTransformers LoRA fine-tuning into LLaMA-Factory, we provide a practical guide for efficient training and deployment of MoE LLMs. KT brings cutting-edge optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization under very low GPU memory. LLaMA-Factory offers a friendly, unified interface.

This integration (akin to Unsloth-style speedups) means even models with tens to hundreds of billions of parameters can be fine-tuned and deployed with low latency on commodity hardware. You get memory savings, speed-ups, and usability together. We encourage you to try LLaMA-Factory + KT for your next MoE project and follow this guide. Feedback is welcome!

KTransformers SFT Docs

  • Kimi K2.5: PyPI Install, LoRA Training, Resume and Serving (中文)
  • KTransformers × LlamaFactory MoE Fine-Tuning Cookbook
  • KTransformers × LlamaFactory MoE Fine-Tuning Cookbook (中文)
  • v0.6.1 Quick Start
  • Fine-Tuning User Guide
  • KT-FT Fine-Tuning and Inference Loop
  • DeepSeek-V4-Flash LoRA quickstart
  • DeepSeek-V4-Flash MXFP4 LoRA training and static deployment
  • Developer Technical Notes
  • DPO Tutorial
  • Injection Tutorial

KTransformers × LlamaFactory MoE Fine-Tuning Cookbook

Research paper: KTransformers-FineTune (KT-FT) · PDF.

From Qwen3.5 to DeepSeek-V4, Kimi-K3, and GLM-5.2, each new ultra-large open model brings a major leap in capability and scale. However, the cost of high-end GPUs still prevents many researchers and developers from fine-tuning these models under constrained resources. KTransformers and LlamaFactory provide a practical alternative: with 1–4 RTX 4090 GPUs and a CPU platform with sufficient memory, users can fine-tune trillion-parameter MoE (Mixture of Experts) models such as the DeepSeek-V3/V4 family, Kimi-K2.5, and GLM-5.2.

KTransformers integrates with LlamaFactory while preserving its familiar training workflow. LlamaFactory remains the unified configuration and orchestration layer for data processing, training, LoRA injection, and inference integration. The GPU runs Attention and Shared Expert modules, while KTransformers manages Routed Experts in CPU memory for heterogeneous GPU–CPU execution.

LlamaFactory and KTransformers MoE fine-tuning architecture

Technical overview: KTransformers places compute-intensive Attention and shared modules on the GPU, while storing the large, sparsely activated Routed Experts in CPU memory. This avoids repeatedly transferring large Expert weights over PCIe and lets the CPU runtime select an optimized backend according to the weight format and available instruction set.

KTransformers MoE model and heterogeneous hardware mapping

Guide at a glance: For a first run, start with native BF16 + LoRA or full fine-tuning + kt_backend: auto. This path requires no weight conversion and starts with two YAML files. If the model provides a native FP8 checkpoint and the training method is LoRA, KTransformers can load the native FP8 Expert weights directly to reduce host-memory usage. Use converted INT8 or AMXINT4 weights only when host memory remains the limiting resource and the matching converted artifacts are available.

Contents

  • Check hardware and install
  • Choose a fine-tuning setup
  • Custom YAML: advanced settings
  • Performance and resource estimates
  • Appendix: complete configurations and Q&A

1. Check hardware and install

1.1 Check hardware resources

Check the CPU instruction set first to identify the available acceleration backend:

1
lscpu | grep -i -E 'Model name|Socket|NUMA|avx512|amx'

Then check GPU memory, host memory, and disk capacity:

1
2
3
nvidia-smi
free -h
df -h /data

Confirm two requirements:

  • Instruction-set compatibility: Native FP8 requires a compatible AMD/x86 CPU with AVX512F, AVX512_BF16, AVX512_VNNI, and AVX512_VBMI. For INT8, auto can select an AMX-INT8 or AVX512-VNNI implementation. AMXINT4 requires AMX and matching converted weights. Native BF16 performs best when AMX is available.
  • Sufficient capacity: Host memory must hold the Expert weights and leave room for activations, gradients, and optimizer states. Disk capacity must also cover checkpoints and temporary files.

1.2 Install the environment

Use a clean Python 3.11 environment. Pin PyTorch 2.9.1 before installing LlamaFactory, and install the KT dependencies last so standard transformers or accelerate packages do not overwrite the KT variants:

1
2
3
4
5
6
7
8
9
10
conda create -n kt-sft python=3.11 -y
conda activate kt-sft

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory

python -m pip install torch==2.9.1 torchaudio==2.9.1 torchvision==0.24.1
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"

After installation, check dependency consistency and the versions that Python actually imports:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
python -m pip check

python - <<'PY'
from importlib.metadata import version

import accelerate
import ktransformers
import kt_kernel
import torch
import transformers

for name, module in {
    "torch": torch,
    "transformers": transformers,
    "accelerate": accelerate,
    "kt_kernel": kt_kernel,
    "ktransformers": ktransformers,
}.items():
    print(f"{name:14s} {getattr(module, '__version__', 'unknown')}")

print(f"{'sglang-kt':14s} {version('sglang-kt')}")
print(f"{'transformers-kt':14s} {version('transformers-kt')}")
print(f"{'accelerate-kt':14s} {version('accelerate-kt')}")

from accelerate.utils.dataclasses import KTransformersPlugin  # noqa: F401
PY

The compatible environment should include these versions:

1
2
3
4
5
6
7
8
9
No broken requirements found.
torch          2.9.1
transformers   5.6.0
accelerate     1.14.0
kt_kernel      0.7.0
ktransformers  0.7.0
sglang-kt      0.7.0
transformers-kt 5.6.0.post2
accelerate-kt  1.14.0.post2

2. Choose a fine-tuning setup

Choose the Expert weight format first, then select LoRA or full fine-tuning. After making both choices, use the corresponding training YAML and Accelerate YAML below.

2.1 Choose the Expert weight format

Native versus converted precision describes the Expert weight format. AMX (Advanced Matrix Extensions) and AVX512 (Advanced Vector Extensions 512-bit) describe CPU instruction sets and backends; they are a separate dimension.

Weight option When to use it Requirements and tradeoffs
Native BF16 checkpoint Host memory is sufficient and a stable, direct training path is the priority No weight conversion is required. When AMX is available, auto selects the currently fastest AMXBF16 path
Native FP8 (8-bit floating point) The model provides a native FP8 checkpoint and should remain in its original weight format Currently used for LoRA with frozen base Experts. Requires the complete AVX512 FP8 extension set; AMX does not support this native FP8 path
KT-converted INT8 Native Expert weights do not fit in host memory Requires a mutually matched pair of INT8 Routed Expert weights and a BF16 non-expert cache. auto can select AMX-INT8 or AVX512-VNNI
KT-converted AMXINT4 Memory pressure is more severe and the additional accuracy risk is acceptable Requires AMX and matching AMXINT4 weights. Validate quality on a representative evaluation set

Prefer native BF16 or native FP8 when host memory allows. Use INT8 or AMXINT4 only when matching converted weights are available and reducing Expert memory is necessary.

2.2 Choose LoRA or full fine-tuning

Expert weight format and parameter-update scope are separate configuration dimensions, but the supported combinations have clear boundaries. Native BF16 supports both LoRA and full fine-tuning. Native FP8, INT8, and AMXINT4 recipes use LoRA with frozen base Experts. Full fine-tuning uses BF16.

Method When to use it Resources and output
LoRA (Low-Rank Adaptation) Recommended by default for rapid task adaptation, repeated experiments, or constrained resources Lower GPU-memory, host-memory, and disk cost. Produces a LoRA adapter and KT Expert LoRA artifacts
Full fine-tuning The task requires updating all target parameters and can afford the higher training and checkpoint cost Requires more GPU memory, host memory, and disk capacity. Produces a complete model checkpoint

2.3 Responsibilities of the two YAML files

Training YAML, Accelerate YAML, and KTransformers execution

Each run uses two YAML files. The training YAML defines the task and owns every KTransformers setting. The Accelerate YAML defines only distributed execution and FSDP2. Current LlamaFactory rejects kt_config in the Accelerate YAML.

File Responsibility Common fields
Training YAML Model, data, LoRA or full fine-tuning, batch, sequence length, output directory, and all KT settings model_name_or_path, dataset, finetuning_type, lora_*, cutoff_len, output_dir, use_kt, kt_cpu_activation, kt_weight_path, kt_non_expert_weight_path, kt_config
Accelerate YAML GPU processes, FSDP2 (Fully Sharded Data Parallel 2), and global mixed precision num_processes, mixed_precision, fsdp_config

Check three relationships before launch:

  1. The number of GPUs in CUDA_VISIBLE_DEVICES must equal num_processes.
  2. If kt_config.kt_model_max_length is set manually in the training YAML, it must cover cutoff_len plus the runtime token margin.
  3. Write the LoRA rank only at the top level of the training YAML. LlamaFactory derives the internal KT fields. Remove LoRA-only fields for full fine-tuning.

2.4 Base YAML for four common setups

Copy the complete configurations from Appendix A.1 and A.2, then replace the fields shown for the selected setup. Each block labels the training YAML and Accelerate YAML sections. kt_config always belongs in the training YAML.

2.4.1 Native BF16 + LoRA
1
2
3
4
5
6
7
8
9
10
11
12
13
# File 1: training YAML
model_name_or_path: /data/models/Your-BF16-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# Do not set kt_weight_path for native BF16
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2
2.4.2 Native FP8 + LoRA
1
2
3
4
5
6
7
8
9
10
11
12
13
# File 1: training YAML
model_name_or_path: /data/models/Your-Native-FP8-Model
finetuning_type: lora
lora_rank: 8
use_kt: true
# Do not set kt_weight_path for native FP8
kt_config:
  kt_backend: auto
  kt_expert_weight_format: fp8

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2
2.4.3 INT8 / AMXINT4 + LoRA
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# File 1: training YAML
finetuning_type: lora
lora_rank: 8
use_kt: true
# INT8 setup
kt_weight_path: /data/models/Your-Routed-Experts-INT8
kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
kt_config:
  kt_backend: auto
  kt_expert_weight_format: int8
  kt_weight_lifecycle: persistent

# For AMXINT4, use matching paths and replace kt_config above with:
# kt_weight_path: /data/models/Your-Model-AMXINT4
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16
# kt_config:
#   kt_backend: AMXINT4

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2
2.4.4 Native BF16 + full fine-tuning
1
2
3
4
5
6
7
8
9
10
11
12
# File 1: training YAML
finetuning_type: full
learning_rate: 1.0e-5
use_kt: true
# Remove lora_rank, lora_alpha, lora_dropout, lora_target, and other LoRA fields
kt_config:
  kt_backend: auto
  kt_expert_weight_format: bf16

# File 2: Accelerate YAML
mixed_precision: bf16
num_processes: 2

2.5 Launch

1
CUDA_VISIBLE_DEVICES=0,1 accelerate launch --main_process_port 0 --config_file qwen35_fsdp2_2gpu.yaml src/train.py qwen35_397b_bf16_lora.yaml

3. Custom YAML: advanced settings

After a base setup launches successfully, tune the following parameters according to the training objective and resource bottleneck. Keep the recommended values unless there is a clear reason to change them, and avoid changing several parameter groups at once.

(1) LoRA capacity: lora_rank, lora_alpha, lora_target, lora_dropout

Change these four fields only at the top level of the training YAML. LlamaFactory automatically derives the rank, alpha, and dropout used internally by KTransformers. Do not repeat them in kt_config or the Accelerate YAML. Start with lora_rank: 8, lora_alpha: 16, lora_target: all, and lora_dropout: 0.0.

If task adaptation is insufficient, increase the rank to 16 and then 32 while keeping alpha at about twice the rank. If a small dataset shows clear overfitting, increase dropout to 0.05.

(2) Sequence length: cutoff_len

Set this field in the training YAML according to the effective token length of most samples. Do not use the maximum dataset length solely for a small number of long samples. Longer sequences increase activation and KT buffer usage, which raises both host-memory and GPU-memory requirements.

The system derives the KT model-length capacity from cutoff_len. To increase it manually, set kt_model_max_length in the training YAML’s kt_config. It must be greater than cutoff_len and include room for additional runtime tokens.

(3) Batch: per_device_train_batch_size, gradient_accumulation_steps

Set both fields in the training YAML. For ultra-large MoE models, start with per_device_train_batch_size: 1. Increase the micro batch only after confirming sufficient GPU and host memory. If GPU memory is insufficient, keep the micro batch at 1 and use gradient accumulation to increase the effective batch.

1
effective batch size = per_device_train_batch_size × number of GPUs × gradient_accumulation_steps

(4) Activation recomputation and CPU Activation Reuse: disable_gradient_checkpointing, kt_cpu_activation

In the training YAML, disable_gradient_checkpointing: false enables gradient checkpointing to trade additional computation for lower activation memory. Setting it to true retains more intermediate results and may improve speed, but uses more memory.

When host memory is sufficient, add this top-level field to the same training YAML:

1
kt_cpu_activation: retain

This option retains CPU Expert activations during checkpoint recomputation, using more host memory to reduce repeated CPU work. When gradient checkpointing is enabled and the field is omitted, CPU and GPU activations are recomputed. kt_cpu_activation controls CPU Expert activation retention, while disable_gradient_checkpointing controls the overall checkpoint-recomputation policy; do not treat them as one switch.

(5) CPU Expert reuse and threads: kt_share_backward_bb, kt_num_threads

Both fields belong in the training YAML’s kt_config. Keep kt_share_backward_bb: true to reuse the buffer required by CPU Expert backward computation. It is separate from kt_cpu_activation; do not disable it unless the configuration for the current model explicitly requires that change.

Set kt_num_threads to the number of physical CPU cores available to the job, not the total number of logical cores including SMT threads. If data preprocessing shares the same CPUs, reserve cores for the DataLoader and system processes.

The system derives kt_threadpool_count from the NUMA topology. Change it manually only after confirming the CPU socket and NUMA layout.

(6) Resume from a checkpoint: resume_from_checkpoint

Set this field in the training YAML to the target checkpoint-* directory. Omit it for a new run. Keep output_dir pointed at the output directory for the current job, never at the base-model directory, and keep overwrite_output_dir: false to avoid overwriting existing results.

4. Performance and resource estimates

The table below uses a 2K context (sequence length = 2048), per-GPU batch size 1, and gradient accumulation 1. It includes only accepted KTransformers BF16 runs that completed real LoRA training. Throughput includes forward, loss, backward, and optimizer steps.

Model and weights GPUs Global batch size Minimum per-GPU memory reference Minimum host-memory reference Fine-tuning throughput (tokens/s)
Qwen3-235B-2507 BF16 LoRA 2 2 ≥ 27.14 GiB ≥ 545.36 GiB 147.91
Qwen3.5-397B BF16 LoRA 8 8 ≥ 20.37 GiB ≥ 1075.68 GiB 215.25

The GPU-memory column reports the highest single-GPU peak from the measured run. The host-memory column reports peak CPU RSS for the training process tree, normalized to GiB. These values are minimum references for the measured configurations, not safety margins. Reserve additional capacity for data loading, caches, checkpoint saving, and system processes.

Results with different GPU counts should not be used to infer linear scaling directly. Increasing cutoff_len, batch size, or KT cache depth increases resource usage. Full fine-tuning also requires additional capacity for gradients, master weights, optimizer states, and complete checkpoints.

Appendix

A.1 Qwen3.5-397B BF16 LoRA training YAML

Save as qwen35_397b_bf16_lora.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
### model

model_name_or_path: /data/models/Qwen3.5-397B-A17B
trust_remote_code: true
disable_gradient_checkpointing: false

### method

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.0
lora_target: all

### dataset

dataset: your_dataset
template: qwen3_5
cutoff_len: 2048
packing: false
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output

output_dir: /data/output/qwen35-bf16-lora
logging_steps: 10
save_strategy: steps
save_steps: 500
plot_loss: true
overwrite_output_dir: false
save_only_model: false
report_to: none

### train

per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### ktransformers

use_kt: true
# kt_cpu_activation: retain  # Optional: use more host memory to reduce CPU Expert recomputation
# kt_weight_path: /data/models/Your-Routed-Experts-INT8  # Converted INT8 Routed Experts
# kt_non_expert_weight_path: /data/models/Your-Non-Expert-Cache-BF16  # Must match the path above
kt_config:
  kt_expert_weight_format: bf16
  kt_backend: auto
  kt_num_threads: 64
  kt_tp_enabled: true
  kt_threadpool_count: 2
  kt_max_cache_depth: 2
  kt_share_backward_bb: true

A.2 FSDP2 Accelerate YAML

Save as qwen35_fsdp2_2gpu.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_version: 2
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

A.3 Q&A

A.3.1 Why can’t a native FP8 checkpoint use the converted INT8 configuration?

The weight formats and loading paths are different. Native FP8 preserves the FP8 weights and scales from the checkpoint and uses the AVX512 native-precision backend. The INT8 recipe reads a mutually matched pair of INT8 Routed Expert weights and a BF16 non-expert cache, then lets auto select an AMX-INT8 or AVX512-VNNI implementation. Do not pass a native FP8 checkpoint through kt_weight_path.

A.3.2 auto did not select the expected backend

Save the lscpu output and startup log, then confirm that the container or virtualization environment did not hide the required instruction set. Native BF16 can select AMXBF16 when AMX is available. Native FP8 should select AVX512. Do not force a backend that the CPU or weight format does not support.

A.3.3 Insufficient host memory

First reduce concurrent data loading, sequence length, and cache depth, and confirm that two copies of the Expert weights are not loaded. If memory is still insufficient, prepare converted INT8 or AMXINT4 weights. For full fine-tuning, estimate the additional memory for gradients, master weights, and optimizer states separately.

A.3.4 GPU OOM

OOM (Out of Memory) means GPU or host memory is insufficient. Reduce cutoff_len, per-GPU batch size, and activation usage first, then increase gradient_accumulation_steps to preserve the global batch size. num_processes, the visible GPU count, and the actual model placement must agree.

A.3.5 kt_model_max_length does not match cutoff_len

When increasing cutoff_len, increase kt_model_max_length as well and leave room for the effective runtime sequence length. A value that is too small causes buffer or shape errors; an unnecessarily large value increases host-memory usage.

Qwen3.5 MoE KT LoRA Serving with SGLang-KT

Last updated: 2026-06-23

This guide documents the current KT-FT loop for Qwen3.5 MoE: train with KT SFT, convert the output once, and serve the fine-tuned result through SGLang with a single merged adapter path.

1
2
3
4
5
6
KT SFT raw output
  -> convert_kt_to_sglang_adapter.py
  -> <MERGED_ADAPTER_DIR>
  -> sglang --lora-paths <name>=<MERGED_ADAPTER_DIR>
  -> server auto-splits expert / non-expert internally
  -> request model=<served_model>:<name>

Training-side KT SFT docs remain separate. This page focuses on the bridge from trained LoRA artifacts to online inference.

1. Scope

Current supported and validated workflow:

  • Base model: Qwen3.5 MoE, for example Qwen3.5-35B-A3B
  • KTransformers version: v0.6.3 or newer
  • KT expert weights: AMX/BF16 SFT-compatible KT CPU expert path
  • User-facing serving input: one converted merged adapter directory
  • Runtime split: expert LoRA goes to the KT CPU expert path; non-expert LoRA goes to SGLang’s LoRA manager. This split happens automatically at server startup.
  • This workflow is for KT MoE expert LoRA artifacts. Standard dense-model PEFT LoRA adapters usually do not need this converter.

2. Artifacts At Each Stage

Raw KT SFT output

After LLaMA-Factory + KT training, the output directory contains two LoRA artifacts:

1
2
3
4
<KT_SFT_OUTPUT_DIR>/
  adapter_model.safetensors      # non-expert LoRA
  fused_expert_lora.safetensors  # expert LoRA in KT fused format
  adapter_config.json

Do not pass this raw directory directly to SGLang serving.

Converted merged adapter

Run the converter once to produce the serving input:

1
2
3
<MERGED_ADAPTER_DIR>/
  adapter_config.json
  adapter_model.safetensors

This merged directory contains both expert and non-expert LoRA tensors in one PEFT-style adapter. Pass only this directory to --lora-paths.

3. Convert Once

1
2
3
4
5
python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --overwrite

Example:

1
2
3
4
5
python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  saves/KT_FT_qwen35B_Moe_custom \
  saves/KT_FT_qwen35B_Moe_custom_sglang \
  --base-model-name-or-path /mnt/data3/models/Qwen3.5-35B-A3B \
  --overwrite

The converter reads fused_expert_lora.safetensors and the existing non-expert adapter_model.safetensors, then writes one merged adapter directory.

If the raw KT SFT output does not contain an adapter_config.json with lora_alpha, pass --lora-alpha <value> explicitly. The converter does not fold LoRA scaling into the tensors; runtime scaling remains lora_alpha / r.

Optional split outputs for debugging:

1
2
3
4
5
6
7
python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  <KT_SFT_OUTPUT_DIR> \
  <MERGED_ADAPTER_DIR> \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --expert-output-dir <EXPERT_ADAPTER_DIR> \
  --nonexpert-output-dir <NONEXPERT_ADAPTER_DIR> \
  --overwrite

For normal serving, only <MERGED_ADAPTER_DIR> is needed.

4. Launch SGLang

Use the KTransformers SGLang fork from this repository and point PYTHONPATH at both kt-kernel/python and third_party/sglang/python.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cd /path/to/ktransformers

PYTHONPATH=/path/to/ktransformers/kt-kernel/python:/path/to/ktransformers/third_party/sglang/python:$PYTHONPATH \
python -m sglang.launch_server \
  --host 127.0.0.1 \
  --port 30006 \
  --model-path /path/to/Qwen3.5-35B-A3B \
  --tokenizer-path /path/to/Qwen3.5-35B-A3B \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-AMXINT4 \
  --kt-method AMXINT4 \
  --kt-cpuinfer 60 \
  --kt-threadpool-count 2 \
  --kt-numa-nodes 0 1 \
  --kt-num-gpu-experts 0 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 2 \
  --max-total-tokens 32000 \
  --served-model-name qwen3.5-kt-ft \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-cuda-graph \
  --disable-custom-all-reduce \
  --enable-lora \
  --lora-backend triton \
  --lora-paths qwen35b_lora=/path/to/KT_FT_qwen35B_Moe_custom_sglang \
  --log-level info

Important points:

  • Pass only one merged adapter through --lora-paths.
  • Do not also pass --kt-expert-lora-path in the normal user workflow.
  • At startup, the server detects the merged KT MoE adapter, splits it internally, and writes runtime cache directories under $TMPDIR/sglang_kt_lora_cache/ (or $SGLANG_KT_LORA_CACHE_DIR if set).
  • Prefer --lora-backend triton for Qwen3.5 full-LoRA generation.

Current constraints:

  • single merged KT composite adapter only
  • --kt-num-gpu-experts 0
  • do not enable --kt-enable-dynamic-expert-update
  • do not use --kt-gpu-prefill-token-threshold
  • --max-running-requests must be at least 2
  • use an AMX/BF16 SFT-compatible KT method such as AMXINT4, AMXINT8, AMXBF16, or BF16

5. Request Semantics

The OpenAI-compatible request model field uses names, not paths.

1
2
--served-model-name qwen3.5-kt-ft
--lora-paths qwen35b_lora=/path/to/merged_adapter

Request behavior in the current single-adapter implementation:

1
2
3
4
5
model=qwen3.5-kt-ft
=> base + KT expert LoRA

model=qwen3.5-kt-ft:qwen35b_lora
=> base + KT expert LoRA + SGLang non-expert LoRA

The suffix after : must match the left-side name in --lora-paths.

If you need a true base-only comparison, launch a separate server without --lora-paths.

6. Smoke Test

1
2
3
4
5
6
7
8
9
curl -sS http://127.0.0.1:30006/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.5-kt-ft:qwen35b_lora",
    "messages": [{"role": "user", "content": "Explain what LoRA is in one sentence."}],
    "temperature": 0.7,
    "max_tokens": 160,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Startup logs should include lines similar to:

1
2
3
Prepared merged KT LoRA adapter ... for runtime: expert=... nonexpert=...
Loaded KT expert LoRA for layer ...
Using triton as backend of LoRA kernels.

7. Advanced: Manual Split Serving

The older split-runtime contract is still available for debugging:

1
2
3
--kt-expert-lora-path <EXPERT_ADAPTER_DIR> \
--enable-lora \
--lora-paths <NONEXPERT_LORA_NAME>=<NONEXPERT_ADAPTER_DIR>

This is not the recommended user-facing path. Normal users should pass one merged adapter directory through --lora-paths only.

8. Troubleshooting

Got LoRA adapter that has never been loaded: lora0

The adapter name in the request must match the left side of --lora-paths. If you launched with qwen35b_lora=..., request model=qwen3.5-kt-ft:qwen35b_lora, not :lora0.

No visible adapter effect

Make sure you are serving the converted merged adapter directory produced by the converter, not the raw KT SFT output directory or a different test adapter.

connection refused

Check that the server is listening on the port you curl, and remember the example above binds to 127.0.0.1, not 0.0.0.0.

Server resolves upstream SGLang instead of this checkout

1
2
3
4
5
python - <<'PY'
import inspect
import sglang.srt.models.qwen3_5 as qwen3_5
print(inspect.getfile(qwen3_5))
PY

The path should come from this repository’s third_party/sglang.

Tutorial: Inject Operator Step by Step

Author: Azure-Tang

TL;DR

This tutorial will guide you through the process of injecting custom operators into a model using the KTransformers framework. We will use the DeepSeekV2-Chat model as an example to demonstrate how to inject custom operators into the model step by step. The tutorial will cover the following topics:

  • TL;DR
  • How to Write Injection Rules
  • Understanding Model Structure
  • Matrix Absorption-based MLA Injection
  • Injection of Routed Experts
  • Injection of Linear Layers
  • Injection of Modules with Pre-calculated Buffers
  • Specifying Running Devices for Modules
  • Muti-GPU
  • How to Write a New Operator and Inject into the Model

How to Write Injection Rules

The basic form of the injection rules for the Inject framework is as follows:

1
2
3
4
5
6
7
8
9
10
- match:
    name: "^model\.layers\..*\.*$"  # Target module name
    class: torch.nn.Linear  # Target module
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda:0"
      # your_op_param_1: 1234
      # your_op_param_2: 5678
  recursive: True
  • match: This field marks the matching rules, which can appear in two forms, name and class. These two matching rules can appear together or separately; they only match when both criteria are met.
  • replace:
    • class: Python class that can be imported to replace the target module. If no replacement is desired, set to default.
    • kwargs: List of parameters needed for module initialization.
      • generate_device: The device for this module, can be set to “cpu”, “cuda”, “cuda:1”, etc.
  • recursive: Whether to recursively inject this module’s submodules, default is True.

For the recursive field: Some modules contain multiple submodules, such as the Self-attention module typically includes q/k/v/o four linear modules. If we replace the self-attention module but do not want the internal linear modules to be covered by other rules, set this rule to False.

Understanding Model Structure

Using deepseek-ai/DeepSeek-V2-Lite-Chat as an example, we can follow the above rules step by step to inject our custom module and run it. KTransformers offers a high degree of flexibility, allowing you to replace/experiment with basic operators. However, it also requires users to clearly understand the structure of the model they are running.

Fortunately, knowing the structure of a model is very simple. Open the file list on the deepseek-ai/DeepSeek-V2-Lite homepage, and you can see the following files:

Inject-Struction

From the .saftensors file, we can see the name of each layer’s weights, corresponding to the match.name attribute in the injection rules. From the modeling_deepseek.py file, we can see the specific implementation of each module class, with the class name corresponding to the match.class attribute in the injection rules.

The structure of the DeepSeekV2 model from the .saftensors and modeling_deepseek.py files is as follows:

Inject-Struction

Supported operators and their corresponding classes are as follows:

match replace backends descriptions
Linear KTransformersLinear KLinearMarlin Marlin as backend
KLinearTorch pytorch as backend
KLinearCPUInfer llamafile as backend
KLinearFP8 Triton fp8_gemm kernel. Requires GPU be able to caluculate fp8 data
experts KTransformersExperts KExpertsTorch pytorch as backend
KExpertsMarlin Marlin as backend
KExpertsCPU llamafile as backend
Attention KDeepseekV2Attention KDeepseekV2Attention MLA implementation
MoE KMistralSparseMoEBlock KQwen2MoeSparseMoeBlock MoE for Qwen2
KDeepseekV2MoE KDeepseekV2MoE MoE for DeepseekV2
Model KQwen2MoeModel KQwen2MoeModel Model for Qwen2
KDeepseekV2Model KDeepseekV2Model Model for DeepseekV2
RoPE RotaryEmbedding RotaryEmbedding RoPE module
YarnRotaryEmbedding YarnRotaryEmbedding RoPE module

Then we start step-by-step injection of custom modules, our targets are:

  • Replace the linear module with custom Marlin linear module.
  • Replace the self-attention module with a custom Absorption-based MLA module.
  • Replace the experts module with a custom Experts module.
  • Replace the MoE module with a custom MoE module.
  • Replace the RoPE module with a custom RoPE module.
  • Set the running device for each module.

The full implementation of the injection rules can be found in the here.

Matrix Absorption-based MLA Injection

For the injection of the Attention module, we only need to use a regular expression to match the module names used in transformers and replace them with our own MLA module implementation. The YAML injection rule is as follows:

1
2
3
4
- match:
    name: "^model\.layers\..*\.self_attn$"  # Regular expression
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # Optimized MLA implementation

As you can see, each rule in the YAML file has two parts: match and replace. The match part specifies the module to be replaced, and the replace part specifies the module to be injected into the model along with the initialization keywords.

Injection of Routed Experts

For Routed Experts (corresponding to the exps in the diagram), the module we inject is CPUInfer, which is wrapped in the wrapper module KTransformersExperts. KTransformersExperts has multiple implementations, and we need to specify keywords to tell the wrapper module which implementation we want to use and how we plan to use it.

In the source code of the transformer, MoE is implemented using nn.ModuleList. We do not want KTransformers to traverse all submodules in the list and inject them one by one, so in this rule, we set recursive: False to prevent recursive injection into the submodules of this module. The YAML rule is as follows:

1
2
3
4
5
6
7
8
9
- match:
    name: "^model\.layers\..*\.mlp\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # Custom MoE kernel with expert parallelism
    kwargs:
      generate_device: "cpu"
      generate_op: "MLPCPUExperts"
      out_device: "cuda"
  recursive: False # Don't recursively inject submodules of this module

If we inject Routed Experts as a custom module, we cannot use the interfaces in the original nn.ModuleList. Therefore, it is necessary to modify the forward function in the FFN module. The simplest method is to implement a new module with a custom forward function and inject it.

1
2
3
4
- match:
    class: ktransformers.models.modeling_deepseek.DeepseekV2MoE
  replace:
    class: ktransformers.operators.experts.KDeepseekV2MoE     # MLP module with custom forward function

Injection of Linear Layers

For the remaining linear layer modules, we aim to use quantized operators to save storage space while improving performance. Since there is no current research on using MLA and quantization together, we do not want to inject linear into the MLA operator. Therefore, we can modify the regular expression and add a type check in the match part of the rule. Only modules that match both the name and class simultaneously will be injected. We also need to pass some keywords similar to the injection of Routed Experts. The YAML rule is as follows:

1
2
3
4
5
6
7
8
- match:
    name: "^model\.layers\.(?!.*self_attn).*$"  # Regular expression
    class: torch.nn.Linear  # Only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # Optimized kernel on quantized data types
    kwargs:
      generate_device: "cuda"
      generate_op: "QuantizedLinearMarlin"

Injection of Modules with Pre-calculated Buffers

To avoid occupying resources when initializing the injected original model, we use torch’s meta device to initialize the original model. The RoPE module pre-calculates some buffers during initialization, but no calculations are performed when using the meta device. Therefore, we need to compensate for the calculation of the buffer when loading the model. Simply, we inject a custom module into the rotary embedding module, which performs pre-calculation during loading. The YAML rule is as follows:

1
2
3
4
- match:
    class: ktransformers.models.modeling_deepseek.DeepseekV2YarnRotaryEmbedding
  replace:
    class: ktransformers.operators.RoPE.YarnRotaryEmbedding

Specifying Running Devices for Modules

Finally, we set a fallback basic attribute generate_device for all modules:

1
2
3
4
5
6
7
8
9
10
11
12
13
- match:
    name: "^model\.layers\..*\.|^lm_head"
  replace:
    class: "default"
    kwargs:
      generate_device: "cuda"

- match:
    name: "^model.embed_tokens"
  replace:
    class: "default"
    kwargs:
        generate_device: "cpu"

Through these two rules, we place all previously unmatched layers (and their submodules) and lm_head on cuda, and the embedding on cpu. Note that the properties of a module will be determined by the first rule it matches. For example, if you later set a new replace.kwargs.generate_device in an injected module, the device set earlier will take precedence. If your computer has multiple cards, you can also configure the model to multiple cards.

Muti-GPU

If you have multiple GPUs, you can set the device for each module to different GPUs. DeepseekV2-Chat got 60 layers, if we got 2 GPUs, we can allocate 30 layers to each GPU. Complete multi GPU rule examples here.

Inject-Struction

First of all, for multi-GPU, we have to inject an new operator KDeepseekV2Model. And set division of the layers to different GPUs. For our case, we have to set the transfer_map in the KDeepseekV2Model operatoras as follows:

1
2
3
4
5
6
7
- match:
    name: "^model$"
  replace:
    class: "ktransformers.operators.models.KDeepseekV2Model"
    kwargs:
      transfer_map:
        30: "cuda:1"

And we have to set the device for each module in the model.

For example, for routed experts, the yaml for one GPU is:

1
2
3
4
5
6
7
8
9
- match:
    name: "^model\.layers\..*\.mlp\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # Custom MoE kernel with expert parallelism
    kwargs:
      generate_device: "cuda:0"
      generate_op: "MLPCUDAExperts"
      out_device: "cuda:0"
  recursive: False # Don't recursively inject submodules of this module

But for two GPUs, we need to set the device for each module in the model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# allcate 0-29 layers‘s out_device to cuda:0
- match:
    name: "^model\.layers\.(0|[1-9]|[12][0-9])\.mlp\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:0"
  recursive: False # don't recursively inject submodules of this module

# allocate 30-59 layers‘s out_device to cuda:1
- match:
    name: "^model\.layers\.([345][0-9])\.mlp\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda:1"
  recursive: False # don't recursively inject submodules of this module

For other modules, we can set the device in the same way.

How to Write a New Operator and Inject into the Model

In this section, we will explain how to write an operator that can be injected, using the implementation of a new linear as an example.

First, all injectable operators need to inherit from the BaseInjectedModule class, which inherits some attributes required by our injection framework. Its initialization function needs to meet the following basic format:

1
2
3
4
5
6
7
8
9
10
11
class LinearTorchInject(BaseInjectedModule):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)

If users have other parameters that need to be passed to this class, they can also be included in the init function and re-passed in the kwargs parameter in the yaml file. For example, if our operator wants to pass a parameter my_param, the init function can be written as:

1
2
3
4
5
6
7
8
9
10
11
12
13
class LinearTorchInject(BaseInjectedModule):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        my_param: bool = True,
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)
        self.my_param = my_param

Then our injection rule can be written as:

1
2
3
4
5
6
7
8
- match:
    name: "^model\.layers\..*$"  # Regular expression matches the module name.
    class: torch.nn.Linear  # Type restrictions can be added.
  replace:
    class: ktransformers.operators.linear.LinearTorchInject  # Inject module path
    kwargs: # Extra parameters
      generate_device: "cuda"
      my_param: True

For the linear module, it is also necessary to read weights from a gguf file. We provide the KLinearBase class to help users read weights from gguf files. Users only need to inherit and implement the load, unload, and forward functions. Therefore, a fully injectable linear class would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
class LinearTorchInject(BaseInjectedModule, KLinearBase):
    def __init__(
        self,
        key: str,
        gguf_loader: GGUFLoader,
        config: PretrainedConfig,
        orig_module: nn.Module = None,
        generate_device: str = "cuda",
        **kwargs,
    ):
        super().__init__(key, gguf_loader, config, orig_module, generate_device, **kwargs)
        KLinearBase.__init__(self)
        self.has_bias = False
        self.dtype = torch.get_default_dtype()
        self.w = None
        self.has_bias = False

    def load(self, w: dict | nn.Parameter | tuple | None = None, device: str|None = None):
        if device is None: device = self.device
        if w is None: w = self.load_weight(device=device)

        if isinstance(w, nn.Parameter):
            self.w = w.to(dtype=self.dtype).view(self.out_features, self.in_features).T
            self.has_bias = False
        elif isinstance(w, tuple):
            self.w = w[0].to(dtype=self.dtype).view(self.out_features, self.in_features).T
            self.bias = w[1].to(dtype=self.dtype)
            self.has_bias = True
        else:
            raise ValueError("Invalid weight type")
        self.w = self.w.to(device)
        if self.has_bias:
            self.bias = self.bias.to(device)

    def unload(self):
        if self.w is not None:
            self.w = None
        if self.has_bias:
            self.bias = None

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        dtype = x.dtype
        out_device = x.device
        x = x.to(device=self.device, dtype=self.dtype)
        x = x @ self.w
        if self.has_bias:
            x = x + self.bias
        x = x.to(dtype=dtype, device=out_device)
        return x

Note that the self.load_weight function is provided by the KLinearBase class to help users load weights from a gguf file into the module. The implementation details of KLinearBase can be found on GITHUB.

KTransformers Fine-Tuning × LLaMA-Factory Integration – Developer Technical Notes

MadSys Lab, KVCache-AI Team, Approaching AI, LLaMA-Factory Team

Introduction

Recent open-source LLMs—from DeepSeek-V3/R1 to Qwen-MoE and Kimi-K2—have surged in performance and scale. Yet due to compute and memory constraints, it is difficult for typical researchers to fine-tune trillion-parameter-class models. We therefore integrate KTransformers with LLaMA-Factory so that, with 2–4 RTX 4090 GPUs and sufficient CPU memory, one can fine-tune ultra-large Mixture-of-Experts (MoE) models such as DeepSeek-671B.

This architecture bridges resource gaps, enabling local fine-tuning of ultra-large models, while also supporting efficient scenario customization at 14B/30B scales. We validate on stylized dialogue, Westernized translation tone, and medical Q&A, achieving rapid adaptation within hours.

Architecturally, LLaMA-Factory orchestrates data/config/training, LoRA insertion, and inference; KTransformers is a pluggable, high-performance operator backend that takes over Attention and MoE under the same training code, enabling GPU+CPU heterogeneity to accelerate training and reduce GPU memory.

image-20251011010558909

We evaluated LoRA fine-tuning with HuggingFace default, Unsloth, and KTransformers backends (same settings and data). KTransformers is currently the only solution feasible on 2–4×24GB 4090s for 671B-scale MoE, and also shows higher throughput and lower GPU memory for 14B MoEs.

Under LoRA (BF16) + NekoQA-10K stylized dialogue HuggingFace Backend Unsloth Backend KTransformers Backend
[14B-DeepSeekV2-Lite] LoRA fine-tuning throughput 303.58 token/s 455.37 token/s 530.38 token/s
[14B-DeepSeekV2-Lite] GPU memory 32.12 GB 9.64 GB 6.08 GB
[671B-DeepSeekV3] LoRA fine-tuning throughput Too Huge to run NOT SUPPORT 40.35 token/s
[671B-DeepSeekV3] GPU memory (sum across GPUs) theoretical 1400 GB † NOT SUPPORT 70 GB †

† The 1400 GB is the theoretical FP16 full-resident footprint (not runnable). 70 GB is the measured peak with KT (Attention on GPU + layered MoE offload).

From the table above, it can be seen that for the 14B model, the KTransformers backend achieves approximately 75% higher throughput than the default HuggingFace solution, while using only about one-fifth of the GPU memory. For the 671B model, both HuggingFace and Unsloth fail to run on a single 4090 GPU, whereas KTransformers is able to perform LoRA fine-tuning at 40 tokens/s, keeping the GPU memory usage within 70 GB.

按照模型划分的对比图_02

Overall View of the KT Fine-Tuning Framework

We detail how KTransformers takes over core operators in LLaMA-Factory’s fine-tuning framework to optimize Attention and MoE.

DeepSeek-V3/V2 MoE models comprise a small-parameter dense Attention part and a large-parameter sparse MoE part. For illustration, consider layer 2 of DeepSeek-V2-Lite-Chat (from which each layer includes both Attention and MoE). Attention compute and KV cache mainly reside on the GPU; the heavyweight MoE part is primarily executed on the CPU. We first cover Attention replacement and inheritance, then MoE encapsulation and backend interfacing, and finally multi-GPU placement.

Attention (LoRA + KT coexist)

KTransformers provides operator injection (BaseInjectedModule), and PEFT provides LoRA layer insertion. For fine-tuning, we design KTransformersLinearLora, inheriting from both KTransformersLinear and LoraLayer:

  • Inheritance: KTransformersLinearLora retains KT’s high-performance paths (prefill_linear/generate_linear) while accepting LoRA parameters (lora_A/lora_B).
  • Replacement: During preparation, we replace original KTransformersLinear layers (Q/K/V/O) with KTransformersLinearLora, preserving KT optimizations while enabling LoRA trainability.

image-20251016182810716

After replacement, LoRA is inserted at Q/K/V/O linear transforms (left), and KTransformersLinearLora contains both KT fast paths and LoRA matrices (right).

image-20251016182920722

MoE (operator encapsulation + backward)

Encapsulation

Given large parameters and sparse compute, we encapsulate the expert computation as a differentiable black-box operator—transparent upstream, replaceable downstream.

  • Upstream (PyTorch graph): we register a custom Autograd Function so the MoE layer appears as a single node. In the left figure (red box), only KSFTExpertsCPU is visible; on the right, the unencapsulated graph expands routing, dispatch, and FFN experts. Encapsulation makes the MoE layer behave like a standard nn.Module with gradients.
  • Downstream (backend): inside the Autograd Function, pybind11 calls C++ extensions for forward/backward. Multiple pluggable backends exist (AMX BF16/INT8; llamafile). The backend can be switched via YAML (e.g., "backend": "AMXBF16" vs. "llamafile").

image-20250801174623919

Backward (CPU)

MoE backward frequently needs the transposed weights $W^\top$. To avoid repeated runtime transposes, we precompute/cache $W^\top$ at load time (blue box). We also cache necessary intermediate activations (e.g., expert projections, red box) to reuse in backward and reduce recomputation. We provide backward implementations for llamafile and AMX (INT8/BF16), with NUMA-aware optimizations.

image-20251016182942726

Multi-GPU Loading/Training: Placement strategy instead of DataParallel

To lower per-GPU memory peaks on 2–4 GPUs, we use model parallelism + explicit placement, not DataParallel (which duplicates the whole model on each GPU).

Key changes:

  1. KTrainer: takes over .to(device) to prevent “move whole model to a single GPU”. Using KT’s optimize-rule YAML, each layer declares device: cuda:0/cuda:1/... and is constructed directly on the target GPU (no extra copies).
  2. Disable automatic DataParallel: when USE_KT=1, we disable automatic DP wrappers from LLaMA-Factory/HF Trainer to avoid duplication and keep full control over sharding.
  3. Gradient aggregation: gradients are reduced to cuda:0. Intermediate activations stay local; only necessary tensors are transferred, cutting communication/activation overhead.

Thus, we keep KT placement strategies under multi-GPU fine-tuning. Users choose a kt_optimize_rule with multi-gpu. For DeepSeek-671B, DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml is a typical 2-GPU plan: KV/attention parts on each GPU; MoE experts sharded on CPU; both GPUs share the workload.

KT-LoRA Fine-Tuning Evaluation

Setup

LLaMA-Factory orchestration, KTransformers backend, LoRA (rank=8, α=32, dropout=0.1, BF16), GAS=16, qlen=512, with the same KT optimize rule as training. We evaluate (a) stylized dialogue transfer and (b) two small-scale representative benchmarks: Translational-Style (generative) and AfriMed-QA (medical vertical; SAQ and MCQ). AMX is enabled; GPUs: 2×48GB RTX 4090; CPU: Intel Xeon Platinum 8488C.

Results

Stylized Dialogue (CatGirl tone)

Dataset: NekoQA-10K. The fine-tuned model consistently exhibits the target style (red boxes) versus neutral/rational base (blue). This shows KT-LoRA injects style features into the generation distribution with low GPU cost.

image-20251016175848143

Translational-Style benchmark (generative)

Dataset: Translational-Style-ChatLLM. Metrics: BLEU-1/2/3/4, ROUGE-1/2/L.

Translational-Style dataset BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L
V2-Lite (no LoRA) 20.66 8.33 4.54 2.89 22.71 4.52 19.19
KT-LoRA fine-tuned V2-Lite 35.41 22.44 15.42 11.18 42.03 18.38 33.10
V3 base (no LoRA) 8.49 3.34 1.62 0.96 15.91 2.55 10.07
KT-LoRA fine-tuned V3 37.02 23.70 16.21 11.49 43.43 18.96 34.54

As shown by the test results in the tables above, under a unified workflow and placement strategy, both model scales exhibit consistent gains after fine-tuning, supporting the usability and effectiveness of the “KT backend + LoRA fine-tuning” combination for generative style control. At the same time, this indicates that KT’s heterogeneous placement and operator optimizations can stably support small-sample adaptation in the style domain.

Medical Vertical Benchmark (AfriMed-SAQ/MCQ)

The dataset adopts AfriMed-QA (ACL 2025), a domain-specific dataset for the medical field in Africa with strong scenario customization characteristics, comprising two formats—multiple-choice questions (MCQ) and short-answer questions (SAQ)—which in this case serve as the evaluation for vertical-domain fine-tuning. In terms of evaluation criteria, BLEU/ROUGE are used for SAQ, and Accuracy is used for MCQ.

AfriMed-QA (SAQ) BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L
V2-Lite (no LoRA) 13.58 11.12 9.10 7.23 22.48 7.81 11.73
KT-LoRA fine-tuned V2-Lite 35.90 27.63 22.99 19.15 35.25 17.50 28.44
V3 base (no LoRA) 12.75 10.27 8.05 5.99 20.33 5.65 10.11
KT-LoRA fine-tuned V3 42.42 34.12 28.95 24.54 41.97 22.37 33.28
AfriMed-QA (MCQ) Accuracy
V2-Lite (no LoRA) 0.0645
KT-LoRA fine-tuned V2-Lite 0.4812
V3 base (no LoRA) 0.5833
KT-LoRA fine-tuned V3 0.7930

As shown in the tables above, (1) DeepSeek-V3 (671B) after KT-LoRA fine-tuning achieves clearly higher performance than the fine-tuned DeepSeek-V2-Lite (14B) on both MCQ and SAQ, and it also surpasses the V3 base model. Within our small-scale setting, this preliminarily indicates that KT-LoRA fine-tuning of ultra-large-parameter models has practical significance in vertical domains.

(2) Across both SAQ/MCQ sub-tasks, KT-LoRA delivers consistent gains, indicating that—with KT’s heterogeneous placement and backend operator support—LoRA fine-tuning can effectively inject the key knowledge points of vertical domains such as medicine into the model.

Limitations

At present, most of our testing is conducted on single datasets and at small scale (≤ 20k examples), with the goal of providing existence evidence of system effectiveness for KT-LoRA fine-tuning, rather than drawing generalized conclusions about algorithmic generalization or scaling laws. Our report primarily presents representative figures; to support stronger algorithmic claims, larger sample sizes, multi-lingual/multi-domain datasets, and multi-seed repeated experiments would be required—these are beyond the scope of this work.

We also warmly welcome everyone to join the open-source LLaMA-Factory KT fine-tuning project. If you have additional test results, we especially welcome you to record them in the shared spreadsheet below, and to include the corresponding kt_optimize_rule files, dataset examples, training/evaluation YAMLs, and detailed GPU-memory and CPU configurations for community reference and reproducibility~!

Speed Tests

End-to-End Performance

Definitions

step_time：time per optimization step (tensor movement + Attention + MoE + others).

tokens_per_step = GAS × qlen；token/s = tokens_per_step / step_time。 We use GAS=16, qlen=512 → tokens_per_step=8192.

Measured

Model step_time (s) tokens/step token/s
DeepSeek-V3-671B 203 8192 40.35
DeepSeek-V2-Lite-14B 36 8192 227.6
MoE Compute (DeepSeek-V3-671B)

Theory

  • MoE per-layer, per-token FLOPs (forward+backward) approx.: $$ \text{FLOPs}_{\text{per-layer, per-token}} \approx c \cdot k \cdot H \cdot I $$

​ with $k = 8$（Top-k），$H = 7168$（hidden size），$I = 2048$（intermediate size），$c\approx16$（≈6 forward + ≈10 backward matmuls）。

  • Per-step across all MoE layers: $$ \text{FLOPs}{\text{per-step}} \approx c \cdot qlen \cdot k \cdot H \cdot I \cdot L{\text{MoE}} $$

​ Plugging $c=16, qlen=512, k=8, H=7168, I=2048, L_{MoE}=58$，$\text{FLOPs}_{\text{per-step}} \approx 55.8\ \text{TFLOPs}$.

Measured (MoE TFLOPS on CPU)

If the MoE-only time per step is t_moe (seconds), $\text{TFLOPS} = \text{FLOPs}_{\text{per-step}} / \text{step_per_second}.$

Use MoE-phase time, not full step_time, to get MoE throughput.

TFLOPS Forward Backward
Average 17.55 18.41

Memory Footprint

  • DeepSeek-V3 (671B; 58 MoE layers out of 61): ~70 GB total GPU, ~1.2–1.3 TB host memory.
  • DeepSeek-V2-Lite (14B; 26 MoE layers out of 27): ~5 GB GPU, ~30 GB host memory.

Conclusion

Integrating KTransformers LoRA with LLaMA-Factory provides a practical path to efficiently train and deploy MoE LLMs. KT contributes placement strategies and operator optimizations (DeepSeek/Qwen/Kimi support with AMX-accelerated kernels), and LoRA enables customization with very low GPU memory; LLaMA-Factory supplies a coherent user-level interface.

This means even tens-to-hundreds-of-billion-parameter MoE models can be fine-tuned and served with low latency on ordinary hardware. The approach balances memory savings, speed, and usability, turning ultra-large models into tools that developers can actually wield.

DPO Training with LLaMA-Factory

This tutorial demonstrates how to use Direct Preference Optimization (DPO) to fine-tune a language model using the LLaMA-Factory framework. DPO is a method for training models based on human preferences, allowing for more aligned and user-centric outputs.

Installation

Step 1: Create a conda environment and suit it for KTransformers

1
2
3
conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13]
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
conda install -y -c nvidia/label/cuda-12.8.0 cuda-runtime

Step 2: Install the LLaMA-Factory environment

1
2
3
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

Step 3: Install KTransformers

Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4

(Note: The CUDA version can differ from that in the wheel filename.)

1
pip install ktransformers-0.4.4+cu128torch28fancy-cp312-cp312-linux_x86_64.whl
Option 2: Install KTransformers from source
1
2
3
4
5
6
git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git
cd ktransformers/kt-sft
export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" # set according to your GPU

pip install -r "requirements-sft.txt"
KTRANSFORMERS_FORCE_BUILD=TRUE pip install -v . --no-build-isolation

Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases

1
2
3
4
5
# abi=True/False can find from below
# import torch
# print(torch._C._GLIBCXX_USE_CXX11_ABI)

pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl

Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton)

1
2
git clone https://github.com/kvcache-ai/custom_flashinfer.git
pip install custom_flashinfer/

Prepare Models

We use deepseek-ai/DeepSeek-V2-Lite as an example here. You can replace it with other models such as Kimi K2.

How to start

1
2
3
4
5
6
# For LoRA SFT
USE_KT=1 llamafactory-cli train examples/train_lora/deepseek2_lora_dpo_kt.yaml
# For Chat with model after LoRA SFT
llamafactory-cli chat examples/inference/deepseek2_lora_dpo_kt.yaml
# For API with model after LoRA SFT
llamafactory-cli api examples/inference/deepseek2_lora_dpo_kt.yaml

For example, we provide the YAML file as follows:

（1）examples/train_lora/deepseek2_lora_dpo_kt.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
### model

model_name_or_path: deepseek-ai/DeepSeek-V2-Lite
trust_remote_code: true

### method

stage: dpo
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
pref_beta: 0.1
pref_loss: sigmoid  # choices: [sigmoid (dpo), orpo, simpo]

### dataset

dataset: dpo_en_demo
template: llama3
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output

output_dir: saves/Kllama_deepseekV2_DPO
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### ktransformers

use_kt: true # use KTransformers as LoRA sft backend
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
cpu_infer: 64
chunk_size: 8192

For more details about –kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md

Then, you can use the lora adapter saved in saves/Kllama_deepseekV2_DPO for inference the same as the sft training. For example,

1
2
3
4
5
6
7
8
9
10
model_name_or_path: DeepSeek-V2-Lite-Chat
adapter_name_or_path: saves/Kllama_deepseekV2_DPO
template: deepseek
infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
trust_remote_code: true

use_kt: true # use KTransformers as LoRA sft backend to inference
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
cpu_infer: 32
chunk_size: 8192

KT-CLI

⚠️ Note: This feature is currently under active development. Many functionalities are not yet complete and are being improved. Please stay tuned for updates.

Design Philosophy

KT-CLI is designed to minimize the burden of reading documentation. Instead of requiring users to read lengthy docs, the CLI provides:

  • Interactive Mode: Run commands without arguments to get step-by-step guided prompts

  • Direct Mode: Pass arguments directly for automation and scripting

    💡 Tip: The arguments are fully compatible with the previous SGLang + KTransformers approach, so you can migrate seamlessly.

Simply run a command, and the CLI will interactively guide you through the process!

Usage

You can check the usage by kt --help

1
kt [OPTIONS] COMMAND [ARGS]...

KTransformers CLI - A unified command-line interface for KTransformers.

Options

Option Description
--help Show this message and exit.

Commands

Command Description
version Show version information
chat Interactive chat with running model
quant Quantize model weights
bench Run full benchmark
microbench Run micro-benchmark
doctor Diagnose environment issues
model Manage models and storage paths
config Manage configuration
sft Fine-tuning with LlamaFactory

GLM-5.3-flash: Native KTransformers Support for 1M Context

1. 1M Context and Native Multimodality

GLM-5.3-flash natively supports a context window of up to 1M tokens. A single request can process a large codebase, a long document, or a long-running agent task without repeatedly splitting the context.

The model also supports images, video, reasoning, and tool calling, and can be used directly by coding agents through an OpenAI-compatible API. GLM-5.3-flash has approximately 321B parameters. Its 45 layers comprise 34 Linear Attention layers and 11 DSA layers.

2. Native-Precision KTransformers Support

KTransformers (KT) reads the official GLM-5.3-flash FP8 weights directly. No model conversion or additional quantization of expert weights is required.

The FP8 model occupies approximately 306 GiB. Reserve at least 350 GB of available system memory. The current implementation supports:

  • NVIDIA SM89 and SM120 GPUs (RTX 40 and 50 series)
  • The AVX-512 FP8 CPU expert kernel
  • Heterogeneous CPU-GPU expert inference and Layerwise Prefill
  • A context window of up to 1M tokens
  • Multimodality: text, multiple images, video
  • Tool calling

3. Installation

Use a clean Python 3.11 environment and run:

1
pip install "ktransformers[sglang]"

This command installs compatible versions of KT Kernel, SGLang-KT, and Transformers-KT.

4. Launch

Replace /path/to/GLM-5.3-flash with the model directory. The configurations below enable Layerwise Prefill, multimodal input, and decode CUDA Graphs by default. The model supports up to 1M context; the examples use the validated 501025-token configuration.

4.1 Four-GPU Launch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 4 \
  --context-length 501025 \
  --mem-fraction-static 0.60 \
  --chunked-prefill-size 4096 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 40 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

4.2 Single-GPU Launch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
MODEL_PATH=/path/to/GLM-5.3-flash

CUDA_VISIBLE_DEVICES=0 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --kt-weight-path "$MODEL_PATH" \
  --served-model-name GLM-5.3-flash \
  --host 0.0.0.0 \
  --tp-size 1 \
  --context-length 501025 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 2048 \
  --kt-method FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 0 \
  --kt-gpu-prefill-token-threshold 2048 \
  --cuda-graph-bs 1 2 4 \
  --limit-mm-data-per-request '{"image":8,"video":1}' \
  --mm-process-config '{"image":{"max_pixels":1254400}}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

When Layerwise Prefill is enabled for GLM-5.3-flash, the current implementation normalizes the resident GPU expert count to zero. The --kt-num-gpu-experts 40 setting in the four-GPU example therefore does not keep 40 experts resident during Layerwise Prefill.

The server listens on http://localhost:30000 by default. Check the endpoint after startup:

1
curl http://localhost:30000/v1/models

The OpenAI-compatible endpoint is:

1
http://localhost:30000/v1/chat/completions

5. Multimodal Request Boundaries

  • A request may contain text only, or text with up to eight images.
  • A request may instead contain text with one video.
  • Images and video cannot appear in the same request.
  • Video uses the regular mixed-prefill path rather than Layerwise Prefill. Decode CUDA Graphs remain supported.

Running KTransformers on AVX2 CPUs

This tutorial explains how to run KTransformers on machines that only support AVX2 (without AVX512 or AMX).

Table of Contents

  • Supported Precision Formats
  • Hardware Requirements
  • Installation
  • Verification
  • Starting the Inference Server
    • Example: Qwen3-30B-A3B (BF16)
    • Example: Qwen3.5-35B-A3B-FP8 (FP8)
    • Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)
    • Example: Kimi-K2.5 (RAWINT4)
    • Sending Requests
  • Performance Tuning
  • FAQ

Supported Precision Formats

--kt-method Precision Description
BF16 BF16 native precision Zero precision loss, uses BF16 weights directly
FP8 FP8 block quantization
GPTQ_INT4 INT4 GPTQ
RAWINT4 Raw INT4 with BF16 scales Used by Kimi-K2.5; weights stored in compressed SafeTensor format

Hardware Requirements

  • CPU: x86-64 + AVX2 + FMA (Intel Haswell 2013+ / AMD Zen+)
  • GPU: NVIDIA 24GB+ VRAM (RTX 3090/4090/5090, etc.)
  • Memory: At least the size of the model weights (e.g., Qwen3-30B-A3B BF16 requires 64GB+)
  • OS: Linux

Installation

Build and install from source (one-click install for kt-kernel + SGLang):

1
2
3
4
5
6
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive

# One-click install
./install.sh

On AVX512 or AMX machines, you can also manually force AVX2 compilation:

1
2
3
4
export KT_RAWINT4_BACKEND=avx2
export CPUINFER_CPU_INSTRUCT=AVX2
export CPUINFER_ENABLE_AMX=OFF
./install.sh kt-kernel --manual

Verification

1
2
3
4
5
6
7
8
9
# Check if the CPU supports AVX2
lscpu | grep -i avx2

# Check the loaded kt-kernel variant
python -c "import kt_kernel; print(kt_kernel.__cpu_variant__)"
# Expected output: avx2

# System diagnostics
kt doctor

Starting the Inference Server

Use --kt-method BF16, FP8, GPTQ_INT4, or RAWINT4. KT-Kernel will automatically detect the CPU and fall back to the AVX2 backend when AVX512/AMX is unavailable.

Example: Qwen3-30B-A3B (BF16)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Download the model
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /path/to/Qwen3-30B-A3B

# Check physical core count and NUMA node count
lscpu | grep -E "^CPU(s)|Thread(s) per core|NUMA node(s)"

# Start the server (adjust kt-cpuinfer and kt-threadpool-count based on your hardware)
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B \
  --kt-weight-path /path/to/Qwen3-30B-A3B \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 32 \
  --kt-method BF16 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.80 \
  --chunked-prefill-size 8192 \
  --max-running-requests 2 \
  --served-model-name Qwen3 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

Example: Qwen3.5-35B-A3B-FP8 (FP8)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Download the model
huggingface-cli download Qwen/Qwen3.5-35B-A3B-FP8 --local-dir /path/to/Qwen3.5-35B-A3B-FP8

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-weight-path /path/to/Qwen3.5-35B-A3B-FP8 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method FP8 \
  --kt-gpu-prefill-token-threshold 400 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

Example: Qwen3-30B-A3B-GPTQ-Int4 (GPTQ_INT4)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Download the model
huggingface-cli download Qwen/Qwen3-30B-A3B-GPTQ-Int4 --local-dir /path/to/Qwen3-30B-A3B-GPTQ-Int4

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-weight-path /path/to/Qwen3-30B-A3B-GPTQ-Int4 \
  --kt-cpuinfer 16 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 2 \
  --kt-method GPTQ_INT4 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --max-total-tokens 32000 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --disable-shared-experts-fusion

Example: Kimi-K2.5 (RAWINT4)

Note: The following command is optimized for 4x RTX PRO 6000 Blackwell (96GB each) + AMD Threadripper PRO 5995WX (64 cores, 1 NUMA node) + 256GB RAM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Download the model
huggingface-cli download moonshotai/Kimi-K2.5 --local-dir /path/to/Kimi-K2.5

# Start the server
python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /path/to/Kimi-K2.5 \
  --kt-weight-path /path/to/Kimi-K2.5 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 228 \
  --kt-enable-dynamic-expert-update \
  --kt-method RAWINT4 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.95 \
  --chunked-prefill-size 8192 \
  --max-running-requests 4 \
  --context-length 262144 \
  --enable-mixed-chunk \
  --tensor-parallel-size 4 \
  --enable-p2p-check \
  --disable-shared-experts-fusion

Sending Requests

1
2
3
4
5
6
7
# Interactive chat
kt chat

# OpenAI-compatible API
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3","messages":[{"role":"user","content":"Hello"}],"stream":true}'

Performance Tuning

  • --kt-cpuinfer: set to the number of physical cores
  • --kt-threadpool-count: set to the number of NUMA nodes
  • --kt-num-gpu-experts: higher values reduce CPU load but increase GPU VRAM usage
  • Memory bandwidth is often the bottleneck; high-frequency DDR5 memory helps significantly

FAQ

GPU OOM

  • Reduce --kt-num-gpu-experts, --chunked-prefill-size, --max-total-tokens
  • Lower --mem-fraction-static

For more questions, see FAQ.

Frequently Asked Questions

1. SGLang “Using default MoE kernel config” warning at startup

When using kt-kernel with SGLang, you may see a warning like:

1
2
[2026-05-15 20:31:38] Using default MoE kernel config. Performance might be sub-optimal!
Config file not found at .../fused_moe_triton/configs/...

This warning is expected and can be safely ignored. kt-kernel replaces SGLang’s built-in MoE implementation with its own CPU/GPU hybrid dispatch, so SGLang’s fused-MoE Triton kernel configuration is never used. The warning is emitted by SGLang before kt-kernel takes over MoE execution and has no impact on performance or correctness.

2. Where can I find more help?

Check the existing issues or open a new one.

来自: Ktransformers