#quantization (30 Repositories)
Ranked open-source repositories tagged with #quantization, scored by pull request acceptance likelihood and maintainer engagement velocity.
53.8%
60.0h
30 repositories tagged #quantization
timtoole02/Camelid
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Yamz-Labs/kyojin
Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.
FujitsuResearch/OneCompression
Python package for LLM compression
vllm-project/llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
intel/auto-round
A simple and effective post training quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的后训练量化工具包
open-edge-platform/geti
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).
intel/neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
warpfront/hipfire
RDNA-native LLM inference engine in Rust.
SonySemiconductorSolutions/mct-model-optimization
Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.
ModelCloud/GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
giannisanni/pulsar
SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.
RyanCodrai/turbovec
A vector index built on TurboQuant, written in Rust with Python bindings
Tencent/AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
openvinotoolkit/nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
syv-ai/qwen38-27b-rtx3090
Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks
pytorch/ao
PyTorch native quantization for training and inference
hiyouga/LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
VectorDB-NTU/RaBitQ-Library
An official lightweight library for the RaBitQ algorithm and its applications in vector search.
OpenNMT/CTranslate2
Fast inference engine for Transformer models
cpldcpu/BitNetMCU
Neural Networks with low bit weights on low end 32 bit microcontrollers such as the CH32V003 RISC-V Microcontroller and others
syv-ai/HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
huggingface/optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
VinRobotics/vla.cpp
A unified inference runtime for VLA models.
nunchux-ai/nunchaku
[ICLR2025 Spotlight] SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
nunchux-ai/ComfyUI-nunchaku
ComfyUI Plugin of Nunchaku
nunchaku-ai/ComfyUI-nunchaku
ComfyUI Plugin of Nunchaku
mw00/project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
quantumaikr/quant.cpp
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
qualcomm/aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
hwdsl2/docker-whisper
Docker image for a self-hosted Whisper speech-to-text server with speaker diarization and OpenAI-compatible transcription and translation APIs. Powered by faster-whisper. Supports all Whisper models, NVIDIA GPU (CUDA) acceleration, JSON/SRT/VTT output, SSE streaming, offline mode, and multi-arch (amd64, arm64).