Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
This addresses the challenge of manual hyperparameter tuning in SGD for machine learning practitioners, though it appears incremental as it builds on existing preconditioned SGD methods.
The paper tackles the problem of tuning step sizes in stochastic gradient descent by proposing an adaptive approach based on traceable Lipschitz constants and local variance, resulting in a nearly hyperparameter-free algorithm with provable convergence and adaptive behavior on image classification tasks.
This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product.