Relating Reinforcement Learning to Dynamic Programming-Based Planning
This paper is an incremental contribution to the theoretical understanding of the relationship between RL and dynamic programming for researchers in optimal control and machine learning.
This paper explores the relationship between reinforcement learning (RL) and dynamic programming-based planning, which both address sequential decision-making. It develops a derandomized version of RL and provides mathematical analysis on the equivalence of cost minimization and reward maximization, goal termination and infinite-horizon learning, and conditions for discounting to cause goal achievement failure.
This paper bridges some of the gap between optimal planning and reinforcement learning (RL), both of which share roots in dynamic programming applied to sequential decision making or optimal control. Whereas planning typically favors deterministic models, goal termination, and cost minimization, RL tends to favor stochastic models, infinite-horizon discounting, and reward maximization in addition to learning-related parameters such as the learning rate and greediness factor. A derandomized version of RL is developed, analyzed, and implemented to yield performance comparisons with value iteration and Dijkstra's algorithm using simple planning models. Next, mathematical analysis shows: 1) conditions under which cost minimization and reward maximization are equivalent, 2) conditions for equivalence of single-shot goal termination and infinite-horizon episodic learning, and 3) conditions under which discounting causes goal achievement to fail. The paper then advocates for defining and optimizing truecost, rather than inserting arbitrary parameters to guide operations. Performance studies are then extended to the stochastic case, using planning-oriented criteria and comparing value iteration to RL with learning rates and greediness factors.