Training large language models (LLMs) and diffusion models with reinforcement learning used to require completely separate pipelines and algorithms. Now, Tencent's Hunyuan team has open-sourced UniRL, a unified RL post-training framework that brings text, vision-language, image, and video generation models under a single training loop. UniRL decouples model architecture from training algorithms, supporting multimodal scenarios like text-to-image and video generation.
For diffusion and flow matching models, the team developed Flow-DPPO. Traditional PPO relies on importance sampling ratios to constrain policy updates, but in the continuous action spaces of diffusion and flow matching, the variance of sampling estimates is too high, often causing the constraints to fail. Flow-DPPO leverages the Gaussian distribution characteristics of flow matching models' per-step policies, directly using KL divergence to constrain policy updates. It also uses an asymmetric divergence mask to prevent the model from drifting too far, maintaining stable convergence over multiple training rounds and mitigating catastrophic forgetting.
For LLMs, the team simultaneously released the DRPO algorithm. Traditional methods use hard clipping to limit model deviation — once a token exceeds a safe range, the gradient is discarded. DRPO introduces an advantage-weighted quadratic regularization term instead of hard clipping, so even when the model deviates from the target distribution, it still receives continuous gradient correction signals. On the distributed engineering side, UniRL integrates inference engines like SGLang and vLLM-Omni, enabling efficient scaling across multi-node clusters.