A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings
This work addresses the challenge of improving agent performance in partially observable navigation tasks, though it is incremental as it builds on existing methods with a specific auxiliary loss.
The paper tackles the problem of reinforcement learning in partially observable spatial environments by proposing a self-supervised auxiliary loss that predicts the temporal order of state pairs to improve spatial reasoning. The result is a 9.6% increase in accumulative episode reward on a gridworld navigation task compared to a baseline.
In this work we explore an auxiliary loss useful for reinforcement learning in environments where strong performing agents are required to be able to navigate a spatial environment. The auxiliary loss proposed is to minimize the classification error of a neural network classifier that predicts whether or not a pair of states sampled from the agents current episode trajectory are in order. The classifier takes as input a pair of states as well as the agent's memory. The motivation for this auxiliary loss is that there is a strong correlation with which of a pair of states is more recent in the agents episode trajectory and which of the two states is spatially closer to the agent. Our hypothesis is that learning features to answer this question encourages the agent to learn and internalize in memory representations of states that facilitate spatial reasoning. We tested this auxiliary loss on a navigation task in a gridworld and achieved 9.6% increase in accumulative episode reward compared to a strong baseline approach.