Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
This work provides a unifying perspective and practical improvements for reward-based fine-tuning of generative models, which is important for aligning these models with human preferences.
The paper unifies many existing reward-based fine-tuning methods for diffusion and flow models under a common framework called reward score matching (RSM), showing that alignment reduces to score matching toward a reward-guided target. The authors develop simpler redesigns that improve alignment effectiveness and compute efficiency across settings with differentiable and black-box rewards.
Reward-based fine-tuning aims to steer a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are motivated by different perspectives such as Soft RL, GFlowNets, etc., we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching toward a reward-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias--variance--compute tradeoffs of existing designs and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler redesigns that improve alignment effectiveness and compute efficiency across representative settings with differentiable and black-box rewards. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space.