Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
This addresses a bottleneck in adapting reinforcement learning to diffusion models for decision-making, offering a more efficient fine-tuning method for applications like robotics and autonomous driving, though it is incremental as it builds on existing diffusion policy frameworks.
The paper tackles the problem of fine-tuning diffusion policies in decision-making tasks like robotics and gaming, where existing reinforcement learning methods struggle due to computational intractability in likelihood estimation; it introduces NCDPO, a framework that reformulates diffusion policies as noise-conditioned deterministic policies, achieving sample efficiency comparable to MLP+PPO and outperforming existing methods in benchmarks.
Diffusion policies, widely adopted in decision-making scenarios such as robotics, gaming and autonomous driving, are capable of learning diverse skills from demonstration data due to their high representation power. However, the sub-optimal and limited coverage of demonstration data could lead to diffusion policies that generate sub-optimal trajectories and even catastrophic failures. While reinforcement learning (RL)-based fine-tuning has emerged as a promising solution to address these limitations, existing approaches struggle to effectively adapt Proximal Policy Optimization (PPO) to diffusion models. This challenge stems from the computational intractability of action likelihood estimation during the denoising process, which leads to complicated optimization objectives. In our experiments starting from randomly initialized policies, we find that online tuning of Diffusion Policies demonstrates much lower sample efficiency compared to directly applying PPO on MLP policies (MLP+PPO). To address these challenges, we introduce NCDPO, a novel framework that reformulates Diffusion Policy as a noise-conditioned deterministic policy. By treating each denoising step as a differentiable transformation conditioned on pre-sampled noise, NCDPO enables tractable likelihood evaluation and gradient backpropagation through all diffusion timesteps. Our experiments demonstrate that NCDPO achieves sample efficiency comparable to MLP+PPO when training from scratch, outperforming existing methods in both sample efficiency and final performance across diverse benchmarks, including continuous robot control and multi-agent game scenarios. Furthermore, our experimental results show that our method is robust to the number denoising timesteps in the Diffusion Policy.