vllm-project/vllm-omni

Your current environment Model: MiniCPM-o 4.5, native three-stage duplex, one visible NVIDIA L20X GPU (driver 570.133.20). Recorded server startup: vLLM 0.31.0, vLLM-Omni 0.1.dev38+g7f7e747fe. Current environment inspection: torch 2.13.0+cu130, transformers 5.14.1. The server emitted a vLLM/vLLM-Omni version-mismatch warning; compatibility has not been ruled out. Endpoint: /v1/realtime?duplex=1; 1 fps video, stack_frames=1, 200 ms audio appends, realtime pacing, 60 s client timeout. Your code version The local git reflog places HEAD at 16707378d673dc99e366afcfbb10213940d9414d before server startup (17:53) and throughout this test (19:00–19:04 on 2026-10-08, Asia/Shanghai). The branch includes local model changes; this is not claimed to be a clean upstream-main reproduction. The exact imported source revision was not recorded in-process. Installed package metadata reports 7f7e747fe39934723786c0c211b9327a8ec1d5d0. The checkout was updated after the test; the attached client and deplo

Motivation Native Qwen/Qwen-Image-2.1 support is merged in #8099. We would like community help to reduce generation latency and memory use, improve serving throughput, and turn the available optimizations into reproducible deployment recipes. Help wanted: contributors with experience in kernels, attention, compilation, batching, quantization, distributed execution, or profiling are welcome. Hardware owners can contribute baseline runs, traces, quality comparisons, or reviews. Please pick a bounded item below and comment with your hardware and proposed scope so we can coordinate work. This follows the community optimization trackers for SenseNova-U1.5 (#7598), Ming-Image (#8134), and MAGI-2 (#7085). The serving goals are: A — Low latency: minimize single-request end-to-end latency and denoising time. B — High throughput: maximize successful images/s under concurrency while reporting p50/p95/p99 latency and failures. C — Memory efficiency and hardware coverage: qualify practical c

Summary Helios (BestWishYsh/Helios-{Base,Mid,Distilled}) is a first-class diffusion video model in vLLM-Omni — a 40-layer chunked DiT (33-frame chunks, multi-term memory history) served through the diffusion pipeline. It is currently single-request only and its per-request path is serial (chunk loop + per-chunk serial VAE decode), so concurrency does not raise aggregate throughput. This RFC proposes optimizing Helios along two distinct serving goals that want different configurations, and lists a concrete, independently-landable set of work items for each. Every item is labeled lossless or lossy (definitions below), and every lossy item names the accuracy gate it must pass before default enablement. This is also a call for help: most items are adopt and validate an existing framework feature on one model, not new infrastructure — good, well-scoped contributions for someone with a GPU. See §8 Call for help. Goals Goal A — ultra-low latency. Single-request / per-request latency,

Motivation. Evidence base: static read of main @ 29d13e331 (Ming-Image support landed in #8021, commit 71125f80a). No benchmark was run — all targets below are proposed, not measured, and "supported" means documented/source-declared. Every item cites a file path, PR, or doc. Lossy vs lossless — definitions. LOSSLESS = output numerically equivalent to the reference path (single-device BF16, default attention backend, no cache / no quantization) — bit-exact, or differing only by FP reduction-order / non-determinism noise; no accuracy gate needed (the alignment harness suffices where numerics move). LOSSY = changes the generated output by design (an approximation) → requires an accuracy gate (metric + reference, stated per item) before default enablement. Where a label is not clean-cut, a one-line nuance is given. Summary Ming-Image (inclusionAI/Ming-Image-0.1-Design T2I + image editing, and -Design-Layer layer decomposition) landed natively in #8021 as a two-stage pipeline: St

Your current environment The output of python collect_env.py Your output of `python collect_env.py` here Your code version The commit id or version of vllm 0.29.0 The commit id or version of vllm-omni f90c267ffb61c79e42f0165476909b995caedd83 🐛 Describe the bug OmniGPUWorkerBase.determine_available_memory uses upstream memory_profiling(), but ignores its authoritative non_kv_cache_memory result and reconstructs the older formula: profiled_usage = ( int(self.model_runner.model_memory_usage) + profile_result.torch_peak_increase + profile_result.non_torch_increase ) available = max(0, self.requested_memory - profiled_usage) Upstream memory_profiling now computes: total_consumed = before_create.free_memory - after_profile.free_memory transient_peak_headroom = after_profile.torch_peak - after_profile.torch_allocated non_kv_cache_memory = total_consumed + transient_peak_headroom The upstream GPU worker uses this result directly, with a separate CUDA graph reserve when