Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
For RL practitioners, this work makes diffusion policies more viable for online RL by addressing the scalability challenge of training without ground-truth data.
The paper introduces adjoint matching, a simulation-free algorithm for training diffusion policies in online RL, achieving competitive performance with significantly reduced computational overhead.
Diffusion policies have recently emerged as a powerful paradigm for representing complex action distributions in reinforcement learning (RL). However, their application to online RL remains limited by the challenge of scalable training in the absence of ground-truth data, where standard optimization techniques such as score matching are not directly applicable. In this work, we introduce a highly efficient algorithm for optimizing diffusion policies by leveraging recent advances in stochastic optimal control. Our approach is based on adjoint matching, which enables simulation-free training and circumvents the need for explicit likelihood estimation or costly backpropagation through the diffusion process. Furthermore, we propose several extensions that improve the robustness and stability of the method in practical settings. Empirical results demonstrate that our approach achieves competitive performance while significantly reducing computational overhead, making diffusion policies more viable for online RL scenarios.