#inference-engine (30 Repositories)
Ranked open-source repositories tagged with #inference-engine, scored by pull request acceptance likelihood and maintainer engagement velocity.
37.3%
29.2h
30 repositories tagged #inference-engine
zhongkaifu/TensorSharp
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
modelplaneai/modelplane
The open source control plane for AI inference
xLLM-AI/xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
openinfer-project/openinfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
insight-platform/Savant
Python Computer Vision & Video Analytics Framework With Batteries Included
ROCm/MIVisionX
AMD MIVisionX is a computer vision toolkit built around a highly optimized, conformant open-source implementation of the Khronos OpenVX™ 1.3.2 specification. As of the 4.0.0 release, MIVisionX ships three components: the AMD OpenVX™ engine, the AMD RPP OpenVX extension, and the RunVX graph executor — across CPU, HIP, and OpenCL backends.
EfficientMoE/MoE-Infinity
PyTorch library for cost-effective, fast and easy serving of MoE models.
ovg-project/kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
nobodywho-ooo/nobodywho
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
truespar/paddock
Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
pylint-dev/astroid
A common base representation of python source code for pylint and other projects
giannisanni/pulsar
SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.
pegainfer-project/pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
dphnAI/sonar
Large-scale LLM inference engine
Zyora-Dev/zse
The inference engine the open-source world built for itself.
SearchSavior/OpenArc
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
qualcomm/ai-hub-models
Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
qualcomm/ai-hub-apps
The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
peonist-ai/halogen-flash-server
The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)
brontoguana/krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
ferrumox/fox
A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.
FareedKhan-dev/kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
buguroo/pyknow
PyKnow: Expert Systems for Python
kigner/audio.cpp-webui
audio.cpp with a full-task WebUI - pure C++ audio-model inference engine powered by ggml. TTS, ASR/STT, VAD, voice conversion, speaker diarization, music generation. No Python dependency.
ulfurinn/wongi-engine
A rule engine written in Ruby.
siliconflow/onediff
OneDiff: An out-of-the-box acceleration library for diffusion models.
FedML-AI/FedML
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
datawhalechina/zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
zhihu/ZhiLight
A highly optimized LLM inference acceleration engine for Llama and its variants.
Adlik/Adlik
Adlik: Toolkit for Accelerating Deep Learning Inference