Learning Achievement Structure for Structured Exploration in Domains with Sparse Reward
This work addresses exploration challenges in sparse-reward reinforcement learning for domains with complex, high-dimensional observations, offering a method that could enhance learning efficiency in such environments, though it appears incremental as it builds on existing achievement-based approaches.
The paper tackles the problem of exploration in reinforcement learning domains with sparse rewards by proposing Structured Exploration with Achievements (SEA), a multi-stage algorithm that learns achievement representations and dependencies from offline data to guide online exploration, and demonstrates its ability to accurately recover achievement structures and improve exploration in hard, procedurally generated environments like Crafter with high-dimensional observations.
We propose Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm designed for achievement-based environments, a particular type of environment with an internal achievement set. SEA first uses offline data to learn a representation of the known achievements with a determinant loss function, then recovers the dependency graph of the learned achievements with a heuristic algorithm, and finally interacts with the environment online to learn policies that master known achievements and explore new ones with a controller built with the recovered dependency graph. We empirically demonstrate that SEA can recover the achievement structure accurately and improve exploration in hard domains such as Crafter that are procedurally generated with high-dimensional observations like images.