A frequency-domain analysis of inexact gradient methods
This work addresses robustness analysis for optimization algorithms, which is incremental as it applies existing frequency-domain methods to derive bounds.
The paper analyzes robustness of gradient-based methods under inexact gradient computations using frequency-domain stability criteria, deriving convergence rates for inexact gradient descent and Triple Momentum Method, and providing improved analytic bounds for Nesterov's accelerated method on strongly convex functions.
We study robustness properties of some iterative gradient-based methods for strongly convex functions, as well as for the larger class of functions with sector-bounded gradients, under a relative error model. Proofs of the corresponding convergence rates are based on frequency-domain criteria for the stability of nonlinear systems. Applications are given to inexact versions of gradient descent and the Triple Momentum Method. To further emphasize the usefulness of frequency-domain methods, we derive improved analytic bounds for the convergence rate of Nesterov's accelerated method (in the exact setting) on strongly convex functions.