Convergence Analysis of Policy Iteration
This provides theoretical guarantees for policy iteration in control problems, which is incremental as it extends existing analysis to multi-step look-ahead.
The paper analyzes the convergence and optimality of policy iteration for adaptive optimal control of nonlinear systems with known dynamics and cost, establishing uniqueness of the Bellman equation solution and comparing convergence speed with value iteration.
Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optimality of the limit functions, which follows from the established uniqueness of the solution to the Bellman equation, are the main results of this study. Furthermore, a theoretical comparison between the speed of convergence of policy iteration versus value iteration is presented. Finally, the convergence results are extended to the case of multi-step look-ahead policy iteration.