#llm-inference (30 Repositories)
Ranked open-source repositories tagged with #llm-inference, scored by pull request acceptance likelihood and maintainer engagement velocity.
58.7%
77.5h
30 repositories tagged #llm-inference
harleyszhang/lite_llama
A light llama-like llm inference framework based on the triton kernel.
feichai0017/loom-infer
Rust-native GPU operator library for LLM inference, built with cuda-oxide
spiceai/spiceai
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
software-mansion/react-native-executorch
Declarative way to run AI models in React Native on device, powered by ExecuTorch.
felladrin/MiniSearch
Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space
NPC-Worldwide/incognide
Explore the unknown, build the future, own your data.
ai-dynamo/dynamo
A Datacenter Scale Distributed Inference Serving Framework
xLLM-AI/xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
tetherto/qvac
Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.
openvinotoolkit/openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
warpfront/hipfire
RDNA-native LLM inference engine in Rust.
lemonade-sdk/lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
beam-cloud/beta9
Ultrafast serverless GPU inference, sandboxes, and background jobs
Avarok-Cybersecurity/atlas
Pure Rust Inference Engine
InternLM/lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
flashinfer-ai/flashinfer
FlashInfer: Kernel Library for LLM Serving
syv-ai/qwen38-27b-rtx3090
Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks
neuron-core/neuron-ai
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.
cactus-compute/cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
ome-projects/ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
kubernetes-sigs/lws
LeaderWorkerSet: An API for deploying a group of pods as a unit of replication
katanemo/plano
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
foldl/chatllm.cpp
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
bentoml/BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
NightMean/OlliteRT
Turn your Android phone into an OpenAI-compatible LLM inference server — Fully local, private and Open Source
Michael-A-Kuykendall/shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
brontoguana/krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
Mobile-Artificial-Intelligence/llama_sdk
lcpp is a dart implementation of llama.cpp used by the mobile artificial intelligence distribution (maid)
mit-han-lab/spatten
[HPCA'21] SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
amitshekhariitbhu/llm-inference-engineering
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.