Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning
This work addresses sample efficiency problems for reinforcement learning practitioners in domains like navigation, though it appears incremental as it builds on existing methods for exploration and auxiliary tasks.
The paper tackles the challenge of improving sample efficiency in reinforcement learning for environments with large state spaces and sparse rewards by combining exploration strategies with auxiliary task learning using General Value Functions (GVFs) and directed exploration, demonstrating performance advantages over competitive baselines in navigation tasks with varying grid sizes.
Improving sample efficiency is a key challenge in reinforcement learning, especially in environments with large state spaces and sparse rewards. In literature, this is resolved either through the use of auxiliary tasks (subgoals) or through clever exploration strategies. Exploration methods have been used to sample better trajectories in large environments while auxiliary tasks have been incorporated where the reward is sparse. However, few studies have attempted to tackle both large scale and reward sparsity at the same time. This paper explores the idea of combining exploration with auxiliary task learning using General Value Functions (GVFs) and a directed exploration strategy. We present a way to learn value functions which can be used to sample actions and provide directed exploration. Experiments on navigation tasks with varying grid sizes demonstrate the performance advantages over several competitive baselines.