On the Linear Convergence of Policy Gradient under Hadamard Parameterization
This provides theoretical guarantees for policy gradient methods in reinforcement learning, addressing convergence issues for researchers and practitioners, but it is incremental as it focuses on a specific parameterization.
The paper tackles the convergence of deterministic policy gradient under Hadamard parameterization in tabular settings, establishing an initial O(1/k) error rate and proving local linear convergence after a constant number of iterations, with the sub-optimal probability contracting after k0 steps.
The convergence of deterministic policy gradient under the Hadamard parameterization is studied in the tabular setting and the linear convergence of the algorithm is established. To this end, we first show that the error decreases at an $O(\frac{1}{k})$ rate for all the iterations. Based on this result, we further show that the algorithm has a faster local linear convergence rate after $k_0$ iterations, where $k_0$ is a constant that only depends on the MDP problem and the initialization. To show the local linear convergence of the algorithm, we have indeed established the contraction of the sub-optimal probability $b_s^k$ (i.e., the probability of the output policy $π^k$ on non-optimal actions) when $k\ge k_0$.