Fast and Stable Diffusion Planning through Variational Adaptive Weighting
This addresses the challenge of stable and efficient training for diffusion planning in offline RL, offering a practical improvement for researchers and practitioners in reinforcement learning.
The paper tackled the problem of high training costs and slow convergence in diffusion models for offline RL by deriving a variationally optimal uncertainty-aware weighting function with a closed-form polynomial approximation, achieving competitive performance with up to 10 times fewer training steps on benchmarks like Maze2D and Kitchen.
Diffusion models have recently shown promise in offline RL. However, these methods often suffer from high training costs and slow convergence, particularly when using transformer-based denoising backbones. While several optimization strategies have been proposed -- such as modified noise schedules, auxiliary prediction targets, and adaptive loss weighting -- challenges remain in achieving stable and efficient training. In particular, existing loss weighting functions typically rely on neural network approximators, which can be ineffective in early training phases due to limited generalization capacity of MLPs when exposed to sparse feedback in the early training stages. In this work, we derive a variationally optimal uncertainty-aware weighting function and introduce a closed-form polynomial approximation method for its online estimation under the flow-based generative modeling framework. We integrate our method into a diffusion planning pipeline and evaluate it on standard offline RL benchmarks. Experimental results on Maze2D and Kitchen tasks show that our method achieves competitive performance with up to 10 times fewer training steps, highlighting its practical effectiveness.