Exploratory State Representation Learning
This addresses the problem of exploration for reinforcement learning agents in complex, reward-free settings, though it is an incremental improvement over existing state representation learning methods.
The paper tackles the challenge of exploration and state representation learning in reward-free environments by proposing XSRL, which jointly learns compact state representations and a discovery policy using a learning progress bonus. Experimental results show that XSRL enables efficient exploration in image-based environments and accelerates reinforcement learning tasks.
Not having access to compact and meaningful representations is known to significantly increase the complexity of reinforcement learning (RL). For this reason, it can be useful to perform state representation learning (SRL) before tackling RL tasks. However, obtaining a good state representation can only be done if a large diversity of transitions is observed, which can require a difficult exploration, especially if the environment is initially reward-free. To solve the problems of exploration and SRL in parallel, we propose a new approach called XSRL (eXploratory State Representation Learning). On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the maximization objective of a discovery policy. This results in a policy that seeks complex transitions from which the trained models can effectively learn. Our experimental results show that the approach leads to efficient exploration in challenging environments with image observations, and to state representations that significantly accelerate learning in RL tasks.