The Phenomenon of Policy Churn
This addresses the understanding of learning dynamics in deep RL for researchers, revealing an overlooked mechanism that could influence exploration strategies.
The paper identifies and studies policy churn, the rapid change of greedy policies in value-based reinforcement learning, showing it occurs in a large fraction of states within a few updates in setups like DQN on Atari, and hypothesizes it as a beneficial form of implicit exploration that reduces the role of ε-noise.
We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts $ε$-greedy exploration in a fresh light, namely that $ε$-noise plays a much smaller role than expected.