Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization
This addresses the need for efficient algorithms in AI applications where both sample size and dimensionality are massive, though it appears incremental as an enhancement to existing doubly stochastic methods.
The paper tackles the problem of large-scale empirical risk minimization for learning tasks by proposing a doubly stochastic algorithm with an accelerating multi-momentum technique, achieving a provably superior convergence rate while reducing memory usage through mini-batch sampling and block coordinate updates.
Nowadays, algorithms with fast convergence, small memory footprints, and low per-iteration complexity are particularly favorable for artificial intelligence applications. In this paper, we propose a doubly stochastic algorithm with a novel accelerating multi-momentum technique to solve large scale empirical risk minimization problem for learning tasks. While enjoying a provably superior convergence rate, in each iteration, such algorithm only accesses a mini batch of samples and meanwhile updates a small block of variable coordinates, which substantially reduces the amount of memory reference when both the massive sample size and ultra-high dimensionality are involved. Empirical studies on huge scale datasets are conducted to illustrate the efficiency of our method in practice.