vllm-project/vllm

This comes from an old comment and was never backed by measurements, so the first task is to establish whether the effect is real at all. It may not be. Runtime throughput was brought in line with native implementations in #47187, but nobody has measured startup time either before or since. Step 1: is there a difference? Time engine init for a model that has both implementations, e.g.: vllm serve Qwen/Qwen3-0.6B --model-impl transformers vllm serve Qwen/Qwen3-0.6B --model-impl vllm Compare time-to-ready across a few runs and a couple of model sizes. Warm vs cold HF cache matters, so control for it. If there is no meaningful difference, say so and close this issue. That is a perfectly good outcome and still worth having measured. Step 2: only if a delta shows up The backend does some work at load time that native implementations do not, so these are plausible starting points rather than known causes: fuser.py — get_fuser traces each module class with fx to match it against the avail

Summary PTX ISA 9.4 (CUDA 13.4.0 Developer Preview) exposes a new ldmatrix variant that performs in-flight sign extension of packed INT4 to INT8 during the shared-memory matrix load: ldmatrix.sync.aligned.m8n16.{x1,x2,x4}{.shared{::cta}}.s8.s4 r, [p]; Source data is densely packed signed 4-bit elements; the hardware expands them to signed 8-bit values in the destination fragment. This can eliminate the register-level INT4→INT8 unpack sequences (lop3/prmt bit tricks and PRMT LUTs) in our W4A8-INT8 kernels, and potentially the dequant-oriented weight interleaving in repack. Doc: PTX ISA 9.4, §9.7.16.5.15 (ldmatrix), https://docs.nvidia.com/cuda/developer-preview/13.4/parallel-thread-execution/index.html Hardware verification (already done) Verified with ptxas/nvdisasm from CUDA 13.4.0 preview (V13.4.46), no GPU required: ptxas accepts the instruction for sm_90a, sm_100a/f, sm_103a, sm_107a/f, sm_110f, sm_120f, sm_121a (PTX targets per doc: sm_90a, sm_100f, sm_110f, sm_120f). SASS i

Summary The NIXL KV connector logs transfer metrics periodically: KV Transfer metrics: Num successful transfers=4, Avg xfer time (ms)=1.381, P90 xfer time (ms)=2.601, Avg post time (ms)=0.672, P90 post time (ms)=0.801, Avg MB per transfer=2.25, Throughput (MB/s)=1629.549, Avg number of descriptors=72.0 Currently there is no documentation explaining what these metrics represent, especially in the context of multi-rank (TP > 1) deployments. This has already caused confusion among users. Current behavior All metrics are aggregated across all TP ranks before summary stats are computed: Each TP rank independently records per-transfer telemetry (transfer_duration, post_duration, bytes_transferred, num_descriptors) via NixlKVConnectorStats.record_transfer() in stats.py. Stats from all ranks are concatenated via aggregate() (list.extend()). reduce() computes averages, percentiles, and throughput over the combined pool of observations from all ranks. This means: "Num successful transfers" i

🚀 The feature, motivation and pitch Available today via FlashInfer 0.6.8, we can leverage this fused MoE Finalize + ResidualAdd + AllReduce + RMSNorm flashinfer-ai/flashinfer#2982. This should be added as a new torch.compile custom pass. I think moe_finalize might still be inside the wrapped fused_moe op so this might require pulling that out. Alternatives No response Additional context No response Before submitting a new issue... Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

🚀 The feature, motivation and pitch The current priority scheduling only supports evicting low-priority requests from the running queue when resources are insufficient. However, when scheduling the waiting queue, if pending requests cannot be scheduled (for example, when the number of requests in the running queue has reached max_num_seqs), even high-priority requests cannot preempt requests in the running queue. Alternatives No response Additional context No response Before submitting a new issue... Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.