Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
This work addresses the problem of distribution shifts in offline-to-online reinforcement learning, which is critical for agents learning from pre-collected data to adapt efficiently in real-world scenarios.
The paper proposes DUAL, an efficient diffusion uncertainty-aware framework for offline-to-online reinforcement learning. DUAL distills a fast-sampling diffusion actor policy and transition model offline, and uses Laplace approximation and distance transition-state-shift detection for uncertainty quantification to improve online exploration versus exploitation, leading to improved online expected return over O2O-RL baselines.
Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distributions. Existing work aims to mitigate the harm of this shift by finetuning the policy on trajectory data sampled from a diffusion model. Inspired by this line of work, we propose DUAL: an efficient \textbf{D}iffusion \textbf{U}ncertainty-\textbf{A}ware framework for offline-to-online reinforcement \textbf{L}earning. DUAL utilizes the prior knowledge of the diffusion model to distill a fast-sampling diffusion actor policy and transition model in the offline phase. DUAL also employs a Laplace approximation and distance transition-state-shift detection, thereby using uncertainty quantification to improve exploration versus exploitation in the online phase. We formally show that our actor loss with the Laplace approximation provides a proxy for a principled estimate of epistemic uncertainty. Empirically, DUAL improves the online expected return over O2O-RL baselines across multiple settings and environments.