Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
This addresses a bottleneck in generalization for reinforcement learning applications, such as simulation-to-real transfer, though it is incremental as it builds on existing methods like PPO and A3C.
The paper tackles the problem of high sample variance in reinforcement learning when training on multiple scenes from the same task by proposing a dynamic value estimation technique that treats the collection as multiple underlying MDPs, resulting in consistent improvements across environments like ProcGen and AI2-THOR.
Training deep reinforcement learning agents on environments with multiple levels / scenes / conditions from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods continue to view this collection of scenes as a single Markov Decision Process (MDP) with a common value function; however, we argue that it is better to treat the collection as a single environment with multiple underlying MDPs. To this end, we propose a dynamic value estimation (DVE) technique for these multiple-MDP environments, motivated by the clustering effect observed in the value function distribution across different scenes. The resulting agent is able to learn a more accurate and scene-specific value function estimate (and hence the advantage function), leading to a lower sample variance. Our proposed approach is simple to accommodate with several existing implementations (like PPO, A3C) and results in consistent improvements for a range of ProcGen environments and the AI2-THOR framework based visual navigation task.