Addressing Function Approximation Error in Actor-Critic Methods
This addresses a critical issue in reinforcement learning for AI agents, though it is incremental as it builds on existing Double Q-learning techniques.
The paper tackled the problem of function approximation errors causing overestimated value estimates and suboptimal policies in actor-critic reinforcement learning methods, and proposed mechanisms to minimize these effects, resulting in outperforming state-of-the-art methods on all OpenAI gym tasks tested.
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.