Improved and Generalized Upper Bounds on the Complexity of Policy Iteration
This provides tighter theoretical guarantees for policy iteration convergence, which is incremental but important for researchers in reinforcement learning and optimization.
The paper tackles the problem of bounding the number of iterations for Policy Iteration algorithms in Markov Decision Processes, showing improved upper bounds: Howard's PI requires at most O(m/(1-γ) log(1/(1-γ))) iterations, and Simplex-PI requires at most O(nm/(1-γ) log(1/(1-γ))) iterations, improving prior results by a factor O(log n).
Given a Markov Decision Process (MDP) with $n$ states and a totalnumber $m$ of actions, we study the number of iterations needed byPolicy Iteration (PI) algorithms to converge to the optimal$γ$-discounted policy. We consider two variations of PI: Howard'sPI that changes the actions in all states with a positive advantage,and Simplex-PI that only changes the action in the state with maximaladvantage. We show that Howard's PI terminates after at most $O\left(\frac{m}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$iterations, improving by a factor $O(\log n)$ a result by Hansen etal., while Simplex-PI terminates after at most $O\left(\frac{nm}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$iterations, improving by a factor $O(\log n)$ a result by Ye. Undersome structural properties of the MDP, we then consider bounds thatare independent of the discount factor~$γ$: quantities ofinterest are bounds $τ\_t$ and $τ\_r$---uniform on all states andpolicies---respectively on the \emph{expected time spent in transientstates} and \emph{the inverse of the frequency of visits in recurrentstates} given that the process starts from the uniform distribution.Indeed, we show that Simplex-PI terminates after at most $\tilde O\left(n^3 m^2 τ\_t τ\_r \right)$ iterations. This extends arecent result for deterministic MDPs by Post & Ye, in which $τ\_t\le 1$ and $τ\_r \le n$, in particular it shows that Simplex-PI isstrongly polynomial for a much larger class of MDPs. We explain whysimilar results seem hard to derive for Howard's PI. Finally, underthe additional (restrictive) assumption that the state space ispartitioned in two sets, respectively states that are transient andrecurrent for all policies, we show that both Howard's PI andSimplex-PI terminate after at most $\tilde O(m(n^2τ\_t+nτ\_r))$iterations.