Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems
This work addresses optimization challenges in machine learning for practitioners, but it is incremental as it builds on existing stochastic dual averaging methods.
The authors tackled the problem of regularized empirical risk minimization in machine learning by proposing two new stochastic gradient methods based on stochastic dual averaging with variance reduction, resulting in sparser solutions and achieving the best known convergence rates among nonaccelerated stochastic gradient methods.
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the existing methods because we do not need to take the average of the history of the solutions. This is favorable in terms of both interpretability and generalization. Moreover, our methods have theoretical support for both a strongly and a non-strongly convex regularizer and achieve the best known convergence rates among existing nonaccelerated stochastic gradient methods.