Novelty Search in Representational Space for Sample Efficient Exploration
This addresses exploration challenges in reinforcement learning for domains with sparse rewards, though it appears incremental relative to existing novelty-based methods.
The paper tackles the problem of sample-efficient exploration in sparse-reward environments by introducing a novelty search method in learned representational space, showing improved sample efficiency over strong baselines on maze and control tasks.
We present a new approach for efficient exploration which leverages a low-dimensional encoding of the environment learned with a combination of model-based and model-free objectives. Our approach uses intrinsic rewards that are based on the distance of nearest neighbors in the low dimensional representational space to gauge novelty. We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space for hard exploration tasks with sparse rewards. One key element of our approach is the use of information theoretic principles to shape our representations in a way so that our novelty reward goes beyond pixel similarity. We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.