On Catastrophic Interference in Atari 2600 Games
This addresses the problem of sample inefficiency in reinforcement learning for researchers and practitioners, providing empirical evidence for a previously speculated issue.
The study tested the hypothesis that catastrophic interference causes sample inefficiency in model-free deep reinforcement learning, finding evidence that interference leads to performance plateaus and degrades policies, and showing that controlling for interference improves performance across various architectures, algorithms, and environments.
Model-free deep reinforcement learning is sample inefficient. One hypothesis -- speculated, but not confirmed -- is that catastrophic interference within an environment inhibits learning. We test this hypothesis through a large-scale empirical study in the Arcade Learning Environment (ALE) and, indeed, find supporting evidence. We show that interference causes performance to plateau; the network cannot train on segments beyond the plateau without degrading the policy used to reach there. By synthetically controlling for interference, we demonstrate performance boosts across architectures, learning algorithms and environments. A more refined analysis shows that learning one segment of a game often increases prediction errors elsewhere. Our study provides a clear empirical link between catastrophic interference and sample efficiency in reinforcement learning.