Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
For researchers in text-to-image alignment, this work provides a principled analysis of divergence choice in GRPO, offering a practical method to improve both performance and diversity.
The paper studies how different f-divergences (forward KL, reverse KL, JS) affect GRPO-style RL for autoregressive text-to-image alignment, finding that JS divergence best balances performance and diversity, achieving strong optimization metrics while maintaining generation diversity on LlamaGen and Janus-7B.
Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.