KTransformers

Active
GitHub Python Apache-2.0

Description

A flexible framework for experiencing heterogeneous LLM inference and fine-tuning optimizations — run large language models efficiently on consumer hardware with kernel-level optimizations.

Key Features

  • CPU-GPU heterogeneous computing for efficient LLM inference and fine-tuning
  • AMX/AVX optimized kernels for INT4/INT8 quantized inference on Intel CPUs
  • MoE optimization with NUMA-aware memory management for Mixture-of-Experts models
  • LLaMA-Factory integration for ultra-large MoE model fine-tuning (DeepSeek-V3/R1)
  • FP8 GPU kernel support achieving 3-28x speedup on single GPU with 24GB VRAM
  • Multi-backend support: Intel AMX, NVIDIA CUDA, AMD ROCm, and Ascend NPU

Use Cases

💡 Running DeepSeek-R1/V3 on consumer hardware with 24GB VRAM
💡 Fine-tuning large MoE models with limited GPU memory using quantization
💡 Integrating with SGLang for production LLM serving at scale
💡 Deploying AI agents on edge devices with CPU-GPU hybrid inference
💡 Research on heterogeneous computing optimizations for LLM workloads

Tags

python llm tools framework

Categories

⚡ Agent Tools

Quick Start

cd kt-kernel
pip install .

# For SFT with LLaMA-Factory:
# cd /path/to/LLaMA-Factory
# pip install -e . && pip install -r requirements/ktransformers.txt
Visit GitHub

Related Projects