#llm-serving (10 Repositories)
Ranked open-source repositories tagged with #llm-serving, scored by pull request acceptance likelihood and maintainer engagement velocity.
38.9%
21.4h
10 repositories tagged #llm-serving
Blackwellboy/model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
helixml/helix
♾️ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude, Codex, Gemini and open models on a full private AI Stack ♾️
NVIDIA/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Ascend
pegainfer-project/pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
alibaba/rtp-llm
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
hpcaitech/SwiftInfer
Efficient AI Inference & Serving
thu-pacman/chitu
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.