Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control
This work addresses the challenge of improving generative models with reward fine-tuning, offering a theoretically-sound solution that is incremental but provides specific gains for AI applications like human preference alignment.
The paper tackled the problem of reward fine-tuning for flow and diffusion generative models by framing it as a stochastic optimal control problem, proving the necessity of a memoryless noise schedule and proposing Adjoint Matching, which outperformed existing methods in consistency, realism, and generalization to unseen reward models while maintaining sample diversity.
Dynamical generative models that produce samples through an iterative process, such as Flow Matching and denoising diffusion models, have seen widespread use, but there have not been many theoretically-sound methods for improving these models with reward fine-tuning. In this work, we cast reward fine-tuning as stochastic optimal control (SOC). Critically, we prove that a very specific memoryless noise schedule must be enforced during fine-tuning, in order to account for the dependency between the noise variable and the generated samples. We also propose a new algorithm named Adjoint Matching which outperforms existing SOC algorithms, by casting SOC problems as a regression problem. We find that our approach significantly improves over existing methods for reward fine-tuning, achieving better consistency, realism, and generalization to unseen human preference reward models, while retaining sample diversity.