Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control
This work addresses efficient adaptation in continuous control for robotics and simulation domains, offering a method that leverages abundant data with minimal annotations, though it is incremental by building on existing alignment and diffusion techniques.
The paper tackles offline reinforcement learning by pretraining generative policies on reward-free behavior data and then fine-tuning them with task-specific Q-values, achieving state-of-the-art performance on the D4RL benchmark and maintaining 95% performance with only 1% of Q-labeled data.
Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then fine-tuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\% of performance and still outperforms several baselines given only 1\% of Q-labelled data during fine-tuning.