#inference (30 Repositories)
Ranked open-source repositories tagged with #inference, scored by pull request acceptance likelihood and maintainer engagement velocity.
73.5%
16.2h
30 repositories tagged #inference
defilantech/LLMKube
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.
google/sam
The Open-Source Agent Mesh
timtoole02/Camelid
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
zhongkaifu/TensorSharp
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
Helldez/BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, CPU only, lossless, on stock llama.cpp
gpustack/gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
llm-d/llm-d-router
llm-d Router: The intelligent entry point for inference requests
mohitsoni48/TurboLLM
Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine in one command. No Electron, no Python, offline-first.
xLLM-AI/xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
NVIDIA/nvcf
Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.
modelplaneai/modelplane
The open source control plane for AI inference
Human-Agent-Society/reef
Infrastructure for continually self‑improving agents
raullenchai/Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
vllm-project/semantic-router
A programmable Mixture-of-Models router for heterogeneous LLM inference
trymirai/uzu
A high-performance inference engine for AI models
vllm-project/vllm-omni
A framework for efficient model inference with omni-modality models
llm-d/llm-d
Achieve state of the art inference performance with modern accelerators on Kubernetes
autonomous-ai/autonomous-grid
Your AI intranet: network the computers you already own for inference and training.
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
sgl-project/sglang-omni
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
LMCache/LMCache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
ultralytics/xview-yolov3
YOLOv3 training, preprocessing, validation, and inference for object detection in xView satellite imagery and the xView detection challenge.
ultralytics/yolov3
PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.
openinfer-project/openinfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
open-edge-platform/dlstreamer
Deep Learning Streamer (DL Streamer) Pipeline Framework is an open-source streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines for the Cloud or at the Edge.
orcasound/orcahello
Real-time AI-assisted killer whale notification system (model and moderator portal) :star:
pmady/keda-gpu-scaler
KEDA External gRPC Scaler for GPU workloads - native NVML metrics via DaemonSet, no Prometheus required
ReactiveBayes/ReactiveMP.jl
High-performance reactive message-passing based Bayesian inference engine