Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
This addresses generalization and domain transfer challenges in RL for applications like simulation-to-real-world tasks, though it is incremental as it builds on existing multi-scene strategies.
The paper tackled the problem of high sample variance in multi-scene reinforcement learning by proposing a dynamic value estimation technique that treats each scene as a distinct MDP and uses sparse attention over multiple value function modes, resulting in significant improvements in final reward scores across OpenAI ProcGen environments.
Training deep reinforcement learning agents on environments with multiple levels / scenes from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods, effectively continue to view this collection of scenes as a single Markov decision process (MDP), and thus learn a scene-generic value function V(s). However, we argue that the sample variance for a multi-scene environment is best minimized by treating each scene as a distinct MDP, and then learning a joint value function V(s,M) dependent on both state s and MDP M. We further demonstrate that the true joint value function for a multi-scene environment, follows a multi-modal distribution which is not captured by traditional CNN / LSTM based critic networks. To this end, we propose a dynamic value estimation (DVE) technique, which approximates the true joint value function through a sparse attention mechanism over multiple value function hypothesis / modes. The resulting agent not only shows significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibits enhanced navigation efficiency and provides an implicit mechanism for unsupervised state-space skill decomposition.