Understanding and Preventing Capacity Loss in Reinforcement Learning
This addresses a key bottleneck in RL for researchers and practitioners, though it is incremental as it builds on existing regularization techniques.
The paper tackles the problem of capacity loss in deep reinforcement learning, where non-stationary targets hinder learning progress, and presents a regularizer called InFeR that improves performance in sparse-reward environments like Montezuma's Revenge.
The reinforcement learning (RL) problem is rife with sources of non-stationarity, making it a notoriously difficult problem domain for the application of neural networks. We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents: \textit{capacity loss}, whereby networks trained on a sequence of target values lose their ability to quickly update their predictions over time. We demonstrate that capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks. We then present a simple regularizer, Initial Feature Regularization (InFeR), that mitigates this phenomenon by regressing a subspace of features towards its value at initialization, leading to significant performance improvements in sparse-reward environments such as Montezuma's Revenge. We conclude that preventing capacity loss is crucial to enable agents to maximally benefit from the learning signals they obtain throughout the entire training trajectory.