Asymptotic Bias of Stochastic Gradient Search
This work addresses the theoretical understanding of bias in stochastic gradient methods, which is crucial for improving optimization algorithms in machine learning and statistics, though it appears incremental as it builds on existing theory.
The paper analyzes the asymptotic bias of stochastic gradient algorithms with biased gradient estimators, deriving tight bounds under mild conditions for high-dimensional nonlinear algorithms, and applies these results to study policy-gradient learning, adaptive population Monte Carlo sampling, and recursive maximum split-likelihood estimation in hidden Markov models.
The asymptotic behavior of the stochastic gradient algorithm with a biased gradient estimator is analyzed. Relying on arguments based on the dynamic system theory (chain-recurrence) and the differential geometry (Yomdin theorem and Lojasiewicz inequality), tight bounds on the asymptotic bias of the iterates generated by such an algorithm are derived. The obtained results hold under mild conditions and cover a broad class of high-dimensional nonlinear algorithms. Using these results, the asymptotic properties of the policy-gradient (reinforcement) learning and adaptive population Monte Carlo sampling are studied. Relying on the same results, the asymptotic behavior of the recursive maximum split-likelihood estimation in hidden Markov models is analyzed, too.