Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Models
This addresses a key bottleneck for improving image quality and human alignment in text-to-image generation, though it appears incremental as it builds on prior noise perturbation approaches.
The paper tackles the problem of inefficient and unstable reinforcement learning in flow-matching models for text-to-image generation by proposing Smart-GRPO, which optimizes noise perturbations through an iterative search strategy. The method improves both reward optimization and visual quality compared to baseline methods.
Recent advancements in flow-matching have enabled high-quality text-to-image generation. However, the deterministic nature of flow-matching models makes them poorly suited for reinforcement learning, a key tool for improving image quality and human alignment. Prior work has introduced stochasticity by perturbing latents with random noise, but such perturbations are inefficient and unstable. We propose Smart-GRPO, the first method to optimize noise perturbations for reinforcement learning in flow-matching models. Smart-GRPO employs an iterative search strategy that decodes candidate perturbations, evaluates them with a reward function, and refines the noise distribution toward higher-reward regions. Experiments demonstrate that Smart-GRPO improves both reward optimization and visual quality compared to baseline methods. Our results suggest a practical path toward reinforcement learning in flow-matching frameworks, bridging the gap between efficient training and human-aligned generation.