Tencent-Hunyuan/UniRL

Summary UniRL's runtime capabilities have expanded faster than several user-facing and package-local README files. This RFC coordinates a community refresh of the main framework repository documentation and splits the work into independent, reviewable tasks. The scope of this issue is Tencent-Hunyuan/UniRL. Changes to the separate unirl-project/unirl documentation site should be tracked independently. Goals Make the top-level README, installation guide, examples index, and package-local READMEs agree with the current code and checked-in recipes. Give every documented entrypoint, algorithm, model, rollout path, and reward/data workflow a traceable source of truth. Let contributors claim one focused task and submit it as an independent PR. Avoid broad rewrites or formatting-only churn outside the claimed task. Sources of truth Use implementation/configuration sources before existing prose: Python/CUDA versions and extras: pyproject.toml Entrypoints and built-in defaults: unirl/train_*

Area Reward / judge Feature request Introduce a two-phase construction path for composite rewards so they can validate scorer capabilities before any inner model is loaded. Today Hydra recursively instantiates nested _target_ values from the leaves upward. For a T2AVCompositeScorer, the order is therefore: instantiate each inner spec; instantiate each inner backend; LocalRewardBackend.__init__ immediately calls _load_model(); instantiate T2AVCompositeSpec and T2AVCompositeScorer; only then validate that the weighted mix contains a prompt-to-video term. As a result, an invalid mix such as ImageBind mode: audio_video + CLAP is rejected before training, but only after ImageBind and CLAP have loaded. It may also fail first on a missing ImageBind dependency or GPU allocation, hiding the more useful configuration error. Desired behavior: validate capability constraints from lightweight configuration/spec data; reject invalid reward graphs before model imports, checkpoint reads, or GPU all

What There is no way to score a reward component without also feeding it into training. A RewardService remote backend runs a panel of reward models and aggregates all of them into the single reward the trainer z-scores into the advantage, so every component that gets scored also shapes the gradient. Searching unirl/reward/ for monitor, probe, log_only, or any zero-weight escape hatch returns nothing. Why that's a gap Detecting reward hacking requires at least one signal the optimiser is not pushing on. With no support for that, the pattern gets approximated with a very small weight — #411 ran an independent VideoPickScore probe at weight: 1e-6 specifically so it would be logged while, in the author's words, being unable to influence the gradient. That approximation is close, but not exact, and the reasons are worth writing down. weighted_sum normalizes by the weight total: if self.aggregation_method == "weighted_sum": total_w = sum(weights) return sum(s * w for s, w in zip(s

What ImageBindRewardScorer defaults to mode="audio_video" in both places that set it: unirl/reward/local/imagebind.py:48 — DEFAULT_MODE = "audio_video" unirl/reward/local/imagebind.py:244 — ImageBindSpec.mode: str = "audio_video" In that mode the prompt never reaches the model: need_text = self._mode in ("text_audio", "text_video", "all") # :195 ... if self._mode == "audio_video": return cos(embeddings[ModalityType.AUDIO], embeddings[ModalityType.VISION]) # :223 So the score is a bare cosine between two streams the policy itself generates. Nothing in it is anchored to what was asked for, which means it can be raised by moving audio and video off-distribution together, with no prompt adherence required. Why it matters #411 measured this. MiniMax-H3 t2va, reward = CLAP 1.0 + ImageBind 1.0 (audio_video) with no visual-quality term, 112 rollouts: Total reward climbed 0.13 -> 0.41. The gain was entirely ImageBind: 0.157 -> 0.34. CLAP, the prompt-anchored component, fell: 0.11

Question I have recently been experimenting with on policy distillation algorithms within the UniRL framework. I noticed that the current main branch already includes a DiffusionOPD implementation and a multi-teacher example. I would like to explore whether UniRL’s on-policy distillation support could be further generalized to accommodate OPD algorithms for LLMs or VLMs. Is there a plan to extend on-policy distillation support in UniRL? If so, I would be happy to contribute an LLM-based OPD implementation to the framework. Thanks!