On the Effectiveness of the z-Transform Method in Quadratic Optimization
This work addresses the problem of understanding and improving optimization algorithms for researchers in machine learning and applied mathematics, but it is incremental as it applies an existing method to known bottlenecks in optimization analysis.
The paper tackled the problem of analyzing optimization algorithms by applying the z-transform method, a classical tool from signal processing, to derive new asymptotic results for gradient descent, Nesterov acceleration, averaging, and stochastic gradient descent, demonstrating its effectiveness and versatility in characterizing convergence behaviors such as spectral dimension.
The z-transform of a sequence is a classical tool used within signal processing, control theory, computer science, and electrical engineering. It allows for studying sequences from their generating functions, with many operations that can be equivalently defined on the original sequence and its $z$-transform. In particular, the z-transform method focuses on asymptotic behaviors and allows the use of Taylor expansions. We present a sequence of results of increasing significance and difficulty for linear models and optimization algorithms, demonstrating the effectiveness and versatility of the z-transform method in deriving new asymptotic results. Starting from the simplest gradient descent iterations in an infinite-dimensional Hilbert space, we show how the spectral dimension characterizes the convergence behavior. We then extend the analysis to Nesterov acceleration, averaging techniques, and stochastic gradient descent.