Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
This addresses the challenge of efficient exploration in visual robot control for continuous action spaces, offering incremental improvements in sample efficiency.
The paper tackles the problem of learning complex robot behaviors through structured exploration by proposing Latent Optimistic Value Exploration (LOVE), which combines latent world models with value function estimation and ensembling to guide deep exploration via optimism, resulting in over 20% improved sample efficiency on average compared to state-of-the-art methods, with over 30% improvement in sparse environments.
Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.