VASE: Variational Assorted Surprise Exploration for Reinforcement Learning
This addresses the problem of efficient exploration in RL for agents in sparse-reward environments, representing an incremental improvement over prior surprise-based methods.
The paper tackles the challenge of exploration in continuous control RL with sparse rewards by introducing VASE, a new surprise-based exploration method using a Bayesian neural network and variational inference, which outperforms existing surprise-based techniques in experiments.
Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variational Assorted Surprise Exploration (VASE). VASE uses a Bayesian neural network as a model of the environment dynamics and is trained using variational inference, alternately updating the accuracy of the agent's model and policy. Our experiments show that in continuous control sparse reward environments VASE outperforms other surprise-based exploration techniques.