Increasing the Action Gap: New Operators for Reinforcement Learning
This work addresses the problem of approximation and estimation errors in reinforcement learning for practitioners, though it appears incremental as it builds on existing operator frameworks.
The paper introduces new optimality-preserving operators for Q-functions, such as the consistent Bellman operator, which increases the action gap to mitigate errors in greedy policies, and demonstrates superior performance in discretized continuous problems and strong potential in 60 Atari 2600 games.
This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird's advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.