Efficient Planning with Latent Diffusion
This work addresses the problem of inefficient planning in offline RL for domains with temporally extended tasks, offering a novel approach that improves flexibility and performance, though it appears incremental by building on latent action spaces and diffusion models.
The paper tackles the challenge of efficient planning in offline reinforcement learning for tasks with delayed sparse rewards by introducing a unified framework for continuous latent action space representation and planning using latent diffusion models. The proposed method, LatentDiffuser, achieves competitive performance on low-dimensional locomotion tasks and surpasses existing methods in higher-dimensional tasks.
Temporal abstraction and efficient planning pose significant challenges in offline reinforcement learning, mainly when dealing with domains that involve temporally extended tasks and delayed sparse rewards. Existing methods typically plan in the raw action space and can be inefficient and inflexible. Latent action spaces offer a more flexible paradigm, capturing only possible actions within the behavior policy support and decoupling the temporal structure between planning and modeling. However, current latent-action-based methods are limited to discrete spaces and require expensive planning. This paper presents a unified framework for continuous latent action space representation learning and planning by leveraging latent, score-based diffusion models. We establish the theoretical equivalence between planning in the latent action space and energy-guided sampling with a pretrained diffusion model and incorporate a novel sequence-level exact sampling method. Our proposed method, $\texttt{LatentDiffuser}$, demonstrates competitive performance on low-dimensional locomotion control tasks and surpasses existing methods in higher-dimensional tasks.