MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
This work provides a versatile optimization framework for machine learning practitioners, but it is incremental as it builds upon existing variance-reduction methods.
The authors introduced MURANA, a generic variance-reduced optimization framework for minimizing sums of smooth functions with a regularizer, which supports strategies like sparse gradients and compressed updates, and demonstrated that its special case ELVIRA improves upon Loopless SVRG.
We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework. However, MURANA also encodes new methods as special cases. We highlight one of them, which we call ELVIRA, and show that it improves upon Loopless SVRG.