sgl-project/sglang-omni

Why The Qwen3-Omni vocoder scheduler uses one hard-coded chunk size for every chunk, including the first: sglang_omni/models/qwen3_omni/components/code2wav_scheduler.py:60 — stream_chunk_size: int = 10, applied uniformly (:152 accumulate gate feeding _decode_and_emit). sglang_omni/models/qwen3_omni/config.py:189 — factory_args={"device": "cuda"} only; the factory (code2wav_scheduler.py:271-289) already accepts stream_chunk_size / left_context_size kwargs, but config never passes them. First-audio latency is bounded below by "wait for a full steady chunk". The shared primitive to fix this already exists — resolve_initial_codec_chunk_frames (sglang_omni/models/tts_streaming.py:11-37) — and is wired for higgs_tts and moss_tts_local, but Code2WavScheduler (a StreamingSimpleScheduler) never uses it. A small first chunk (e.g. 4 frames) with a larger steady chunk (e.g. 25) and 25-frame left context is a sensible starting point, with one seam rule to verify: chunk 1 must use the whol

Why Both encoder stage factories hard-code the batching wait window: sglang_omni/models/qwen3_omni/stages.py:852 (image encoder factory) and stages.py:922 (audio encoder factory), both max_batch_wait_ms=50; factory_args in config.py:56-70 passes only {device, dtype}, so there is no override. Every multimodal request waits up to 50 ms inside the encoder stage just to form a batch, before any compute starts — including a lone request: SimpleScheduler._collect_batch (sglang_omni/scheduling/simple_scheduler.py:101-130) blocks the full window on an empty queue, with no fast path. That sits directly on the TTFT/TTFA critical path. The 50 was chosen so c16 video benchmarks batch together (comment at stages.py:846), but it has never been swept; low-concurrency traffic almost certainly wants a smaller window. Scope Expose max_batch_wait_ms as a config value with the current 50 as default (no behavior change by default). Sweep {0, 1, 5, 10, 50} under the standard multimodal workload at c1/c

Why PR #1101 (open) adds exact-shape CUDA-graph replay for code2wav, enabled by default (+7.8% req/s @c8 on H100). Its benefit depends entirely on incoming shapes hitting captured buckets — capture is limited to B1/T{10,20,30,35} — so a shape drift (different chunk sizes after the initial-chunk work, ragged batches from #1026) degrades to eager. #1101 already counts fallbacks per reason (disabled/ineligible/key_miss) and replays in aggregate, but logs them only once at scheduler startup, before real traffic. What is missing: per-shape granularity and periodic/shutdown reporting, so operators can verify hit rate under real traffic and tune capture buckets from data instead of guesses. Scope Env-gated (e.g. SGLANG_OMNI_TRACE_CODE2WAV_GRAPH=1, matching the SGLANG_OMNI_TRACE_* precedent) extension of #1101's existing counters: per-shape hit count, per-shape eager-fallback count, periodic log line (or dump at shutdown). No behavior change when disabled. Approach Extend #1101's _fallbac

There's a relatively self-contained piece of work that's a great way to get started — if you'd like to join the SGLang Omni dev team, this is a good first thing to try. In short: SGLang Omni computes the WER of our TTS models by transcribing their output with Qwen3-ASR. So Qwen3-ASR's performance directly matters to us — it sits on the evaluation hot path, and CI itself has strong requirements on the Qwen3-ASR model. Making Qwen3-ASR generally faster speeds up every TTS correctness check that depends on it. What we're asking for If you're interested, try optimizing Qwen3-ASR to make it generally faster, and come with your results attached — before/after numbers, the benchmark command you ran, and the hardware you ran it on. Things to keep in mind: Use the profiling tools already in the repo. All the profilers you need are already here. Paired with an AI assistant, they should be easy to put to good use — you don't need to build measurement infra from scratch. Abstract and reus