GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution
This work addresses a specific bottleneck in image super-resolution for applications requiring efficient high-quality generation, but it appears incremental as it builds on existing RL methods like DPO and GRPO.
The paper tackles the problem of one-step generative image super-resolution (ISR) by proposing Group Direct Preference Optimization (GDPO), which integrates reinforcement learning into training and introduces a noise-aware diffusion model with an unequal-timestep strategy; experiments show it effectively enhances performance, though no concrete numbers are provided in the abstract.
Recently, reinforcement learning (RL) has been employed for improving generative image super-resolution (ISR) performance. However, the current efforts are focused on multi-step generative ISR, while one-step generative ISR remains underexplored due to its limited stochasticity. In addition, RL methods such as Direct Preference Optimization (DPO) require the generation of positive and negative sample pairs offline, leading to a limited number of samples, while Group Relative Policy Optimization (GRPO) only calculates the likelihood of the entire image, ignoring local details that are crucial for ISR. In this paper, we propose Group Direct Preference Optimization (GDPO), a novel approach to integrate RL into one-step generative ISR model training. First, we introduce a noise-aware one-step diffusion model that can generate diverse ISR outputs. To prevent performance degradation caused by noise injection, we introduce an unequal-timestep strategy to decouple the timestep of noise addition from that of diffusion. We then present the GDPO strategy, which integrates the principle of GRPO into DPO, to calculate the group-relative advantage of each online generated sample for model optimization. Meanwhile, an attribute-aware reward function is designed to dynamically evaluate the score of each sample based on its statistics of smooth and texture areas. Experiments demonstrate the effectiveness of GDPO in enhancing the performance of one-step generative ISR models. Code: https://github.com/Joyies/GDPO.