Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels
This work addresses the challenge of long-term planning in reinforcement learning from pixels, which is incremental as it builds on hierarchical methods and world models.
The authors tackled the problem of sample-efficient learning in complex environments with high-dimensional state spaces by introducing Forecaster, a deep hierarchical reinforcement learning approach that plans over high-level goals using a temporally abstract world model, and demonstrated its potential in the AntMaze domain for single-task learning and generalization to new tasks.
The ability to plan at many different levels of abstraction enables agents to envision the long-term repercussions of their decisions and thus enables sample-efficient learning. This becomes particularly beneficial in complex environments from high-dimensional state space such as pixels, where the goal is distant and the reward sparse. We introduce Forecaster, a deep hierarchical reinforcement learning approach which plans over high-level goals leveraging a temporally abstract world model. Forecaster learns an abstract model of its environment by modelling the transitions dynamics at an abstract level and training a world model on such transition. It then uses this world model to choose optimal high-level goals through a tree-search planning procedure. It additionally trains a low-level policy that learns to reach those goals. Our method not only captures building world models with longer horizons, but also, planning with such models in downstream tasks. We empirically demonstrate Forecaster's potential in both single-task learning and generalization to new tasks in the AntMaze domain.