Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
This addresses the challenge of cross-task synergy in multimodal AI for researchers and practitioners, though it appears incremental as it builds on existing RL and ULM concepts.
The paper tackles the problem of simultaneously improving generation and understanding capabilities in unified multimodal large language models (ULMs) by introducing CoRL, a co-reinforcement learning framework, resulting in average improvements of 7% on text-to-image generation datasets and 23% on multimodal understanding benchmarks.
This paper presents a pioneering exploration of reinforcement learning (RL) via group relative policy optimization for unified multimodal large language models (ULMs), aimed at simultaneously reinforcing generation and understanding capabilities. Through systematic pilot studies, we uncover the significant potential of ULMs to enable the synergistic co-evolution of dual capabilities within a shared policy optimization framework. Building on this insight, we introduce CoRL, a co-reinforcement learning framework comprising a unified RL stage for joint optimization and a refined RL stage for task-specific enhancement. With the proposed CoRL, our resulting model, ULM-R1, achieves average improvements of 7% on three text-to-image generation datasets and 23% on nine multimodal understanding benchmarks. These results demonstrate the effectiveness of CoRL and highlight the substantial benefit of reinforcement learning in facilitating cross-task synergy and optimization for ULMs. Code is available at https://github.com/mm-vl/ULM-R1.