Unifying Value Iteration, Advantage Learning, and Dynamic Policy Programming
This addresses the need for more robust and unified reinforcement learning algorithms, though it appears incremental as it builds on existing approaches.
The paper proposes generalized value iteration (GVI), a new dynamic programming algorithm that unifies value iteration, advantage learning, and dynamic policy programming, and shows it has performance guarantees that encompass existing algorithms. Numerical experiments in a simple environment suggest GVI is a promising alternative to previous methods.
Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper we propose a new, robust dynamic programming algorithm that unifies value iteration, advantage learning, and dynamic policy programming. We call it generalized value iteration (GVI) and its approximated version, approximate GVI (AGVI). We show AGVI's performance guarantee, which includes performance guarantees for existing algorithms, as special cases. We discuss theoretical weaknesses of existing algorithms, and explain the advantages of AGVI. Numerical experiments in a simple environment support theoretical arguments, and suggest that AGVI is a promising alternative to previous algorithms.