Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
This work addresses a specific problem in reinforcement learning for discrete diffusion models, offering a novel method for researchers and practitioners in multimodal AI, though it appears incremental as it builds on existing techniques like GRPO.
The paper tackled the challenge of optimizing discrete diffusion models with reinforcement learning by introducing MaskGRPO, which enables scalable multimodal reinforcement learning through effective importance sampling and modality-specific adaptations, leading to more stable updates and improved reasoning and generation quality on benchmarks.
Optimizing discrete diffusion model (DDM) with rewards remains a challenge: the non-autoregressive paradigm makes importance sampling intractable and rollout complex, puzzling reinforcement learning methods such as Group Relative Policy Optimization (GRPO). In this study, we introduce MaskGRPO, the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality-specific adaptations. To this end, we first clarify the theoretical foundation for DDMs, which facilitates building an importance estimator that captures valuable token fluctuation for gradient updates. We then delicately tailored the rollout method for visual sequences, which yields diverse completions and reliable optimization gradients. Upon math reasoning, coding, and visual generation benchmarks, MaskGRPO brings more stable and efficient updates, leading to stronger reasoning performance and better generation quality. This study establishes MaskGRPO as a systematic policy optimization approach and the first practical way for discretized visual diffusion.