Stochastic Optimization of Smooth Loss
This work addresses a limitation in stochastic optimization methods for machine learning practitioners by removing the need for inaccessible optimal classifier information, though it appears incremental in nature.
The paper tackles the problem of stochastic optimization with smooth loss by proving a high probability bound instead of an expectation bound and proposing a strategy to tune step sizes without prior knowledge of the optimal classifier, achieving improved theoretical guarantees.
In this paper, we first prove a high probability bound rather than an expectation bound for stochastic optimization with smooth loss. Furthermore, the existing analysis requires the knowledge of optimal classifier for tuning the step size in order to achieve the desired bound. However, this information is usually not accessible in advanced. We also propose a strategy to address the limitation.