Latent World Models For Intrinsically Motivated Exploration
This addresses the problem of exploration in complex, partially observable environments for reinforcement learning, representing an incremental advancement in representation learning and novelty detection.
The paper tackles sparse-reward, partially observable environments by proposing a self-supervised representation learning method for image-based observations, which arranges embeddings based on temporal distance and uses a world model in latent space to guide exploration; it demonstrates significant improvement on Atari benchmark environments compared to prior work.
In this work we consider partially observable environments with sparse rewards. We present a self-supervised representation learning method for image-based observations, which arranges embeddings respecting temporal distance of observations. This representation is empirically robust to stochasticity and suitable for novelty detection from the error of a predictive forward model. We consider episodic and life-long uncertainties to guide the exploration. We propose to estimate the missing information about the environment with the world model, which operates in the learned latent space. As a motivation of the method, we analyse the exploration problem in a tabular Partially Observable Labyrinth. We demonstrate the method on image-based hard exploration environments from the Atari benchmark and report significant improvement with respect to prior work. The source code of the method and all the experiments is available at https://github.com/htdt/lwm.