Analysis of Control Bellman Residual Minimization for Markov Decision Problem
This work addresses a gap in reinforcement learning for control tasks, but it appears incremental as it builds on existing Bellman residual methods.
The paper tackles the problem of policy optimization in Markov decision problems by establishing foundational results for control Bellman residual minimization, which has been scarcely explored compared to policy evaluation methods.
Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.