Adaptive Stochastic Optimization
This work tackles the inefficiency of tuning optimization parameters for machine learning practitioners, but it is incremental as it summarizes existing research and motivates future work.
The paper addresses the problem of non-adaptive stochastic gradient methods in optimization, which require manual tuning for each application, and proposes adaptive methods to potentially achieve significant computational savings in large-scale training.
Optimization lies at the heart of machine learning and signal processing. Contemporary approaches based on the stochastic gradient method are non-adaptive in the sense that their implementation employs prescribed parameter values that need to be tuned for each application. This article summarizes recent research and motivates future work on adaptive stochastic optimization methods, which have the potential to offer significant computational savings when training large-scale systems.