Anderson Acceleration for Reinforcement Learning
This work addresses the problem of slow convergence in reinforcement learning algorithms, potentially benefiting researchers and practitioners in AI, though it appears incremental as it adapts an existing method to a new domain.
The paper applies Anderson acceleration, a fixed-point computation method, to value iteration in reinforcement learning for the first time, showing preliminary experiments that indicate a significant speed up in convergence.
Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement learning. In this paper, we explain briefly what Anderson acceleration is and how it can be applied to value iteration, this being supported by preliminary experiments showing a significant speed up of convergence, that we critically discuss. We also discuss how this idea could be applied more generally to (deep) reinforcement learning.