verl-project/verl-omni

1. Motivation Qwen3-Omni Thinker is already a first-class omni policy in verl-omni. The README currently lists DPO and GSPO as the supported algorithms. DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) is the production-scale GRPO-family recipe in upstream verl for long-CoT / math RL: clip-higher, token-level policy gradient, overlong reward shaping, and dynamic sampling. There is no Thinker DAPO recipe today. The GSPO V1 launcher (examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh) is a sequence-level GSPO setup (policy_loss.loss_mode=gspo, clip_ratio_{low,high}=3e-4/4e-4, loss_agg_mode=seq-mean-token-mean, reward.reward_manager.name=naive). The legacy GSPO YAML sets reward.reward_manager.name=dapo, which only selects the DAPO reward manager (overlong buffer / math parsing). That is not the DAPO algorithm. Supporting DAPO on Thinker is useful because: Users already ask for it as the default long-horizon RL recipe on Qwen3-class models. The Thinker a

Motivation verl-omni sits on top of verl / vllm-omni / diffusers / transformers / peft and carries temporary patches. Each has one of three fates: drop after rebase/bump (fixed upstream; our pin or version floor predates the fix), drop now (fix already inside our pins; for one item, P3, that means an in-pin rewrite rather than a delete), or blocked on upstream (needs a fix we can only suggest — see Tracker §3). Critically, some things that look like patches are intended adaptations — code that exists because verl-omni trains diffusion/omni models that upstream deliberately doesn't cover, and which would survive a perfect upstream (Part II). Mixing those into patch debt produces wrong drop plans, so this ledger separates them. Current tally: 16 patch-debt line items (Part I: 3 immediate, 3 gated on the diffusers bump, 3 on pin rebases, 7 blocked on upstream) · 13 intended adaptations (Part II) · 2 false positives removed from earlier drafts (Part II record). The tracker adds 8 no

Feature request This is a tracking issue for migrating remaining verl-omni models and algorithms onto the verl v1 trainer (TransferQueue + ReplayBuffer), and for proving that each migrated path is performance-competitive and behavior-compatible with the current v0 recipes. It does not redesign the v1 trainer. Architecture RFCs already exist: Area RFC Status Diffusion v1 trainer (sync / separate_async) #249 Done in #296, #315 Omni model base + v1 sync trainer #232 Done in #253, #258, #312 Omni v1 colocate_async / separate_async #320 In progress (#337, #380) Please attach the corresponding PR in the tables below when you start or land a (model, algorithm, mode) path. Keep this issue as the single dashboard. Motivation v1 is the long-term trainer control plane (aligned with upstream verl): TransferQueue, ReplayBuffer, agent-loop workers, and sync / colocate_async / separate_async modes. Today the architecture is in, but recipe coverage is still narrow. Most production exampl

1. Summary Silent fallbacks hide misconfigurations and corrupt data behind "reasonable" defaults. Because RL training only reveals such bugs as abnormal metric curves hours into a run, they are the worst failure mode in this codebase. This RFC proposes the opposite policy: misconfigurations and corrupt data raise immediately with actionable errors. Fallbacks remain only for legitimate cases: optional dependencies when the user left a value unset, explicitly declared optional features, and transient stochastic anomalies — which get visible counters and abort thresholds instead of silence. The policy follows upstream verl's fail-fast design (Section 3), and verl-omni already applies it in several places (Section 2) — this RFC makes it consistent everywhere. 2. Policy Explicit user intent is sacred. If a config value was explicitly set and cannot be honored, raise — explain what was requested, why it fails, and what to change. Unknown values raise. Any enum-like value (arch, algori

1. Summary Generated videos are currently transferred between workers as raw 5-D [B, T, C, H, W] float32 pixel tensors, pickled in full on every Ray hop. This RFC proposes a staged package to cut memory residency and transfer time without changing the reward function or the training objective: Stop shipping pixels to workers that never read them (actor / ref / old-log-prob hops) — ~90% cut on the biggest hop. Use uint8 instead of float32 pixels on the reward/eval hop — 4× with bit-exact scores for current re-encoding scorers. Enable the existing zero-copy serialization path — removes per-hop pickle churn. 2. Motivation Per-sample payloads are large and are copied more often than necessary: One 8s @ 24fps 512×512 video is ~604 MB as float32 RGB (3 × 192 × 512 × 512 × 4 B). A batch of 8 is ~4.8 GB. Measured anchors: verl multimodal DataProto via Ray object store transfers at an effective 0.0302 GB/s (verl RFC #2847). Ray's docs measure a 1 GiB tensor sum at 23.5 s pickled