#llm-serving (17 Repositories)
Ranked open-source repositories tagged with #llm-serving, scored by pull request acceptance likelihood and maintainer engagement velocity.
37.4%
48.0h
17 repositories tagged #llm-serving
helixml/helix
♾️ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude, Codex, Gemini and open models on a full private AI Stack ♾️
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
openinfer-project/openinfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
NVIDIA/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Blackwellboy/model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
pegainfer-project/pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
alibaba/rtp-llm
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
thu-pacman/chitu
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
jjiantong/Awesome-KV-Cache-Optimization
[ACL 2026] Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
uw-syfi/vibesys
Can AI Agents Build Bespoke Systems?
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
peonist-ai/halogen-flash-server
The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)
hpcaitech/SwiftInfer
Efficient AI Inference & Serving
predibase/lorax
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
zhihu/ZhiLight
A highly optimized LLM inference acceleration engine for Llama and its variants.
microsoft/aici
AICI: Prompts as (Wasm) Programs