An Elementary Approach to Convergence Guarantees of Optimization Algorithms for Deep Networks
This work addresses the need for theoretical convergence analysis in deep learning optimization, though it appears incremental as it builds on existing methods with a focus on computational structures.
The paper tackles the problem of obtaining convergence guarantees for optimization algorithms in deep networks using elementary arguments and computations, resulting in a systematic method to compute smoothness constant estimates that govern convergence behavior.
We present an approach to obtain convergence guarantees of optimization algorithms for deep networks based on elementary arguments and computations. The convergence analysis revolves around the analytical and computational structures of optimization oracles central to the implementation of deep networks in machine learning software. We provide a systematic way to compute estimates of the smoothness constants that govern the convergence behavior of first-order optimization algorithms used to train deep networks. A diverse set of example components and architectures arising in modern deep networks intersperse the exposition to illustrate the approach.