Coordinate Descent with Online Adaptation of Coordinate Frequencies
This work addresses the efficiency of training linear models like SVM and LASSO for machine learning practitioners, offering an incremental improvement over existing coordinate descent methods.
The paper tackles the problem of non-uniform coordinate selection in coordinate descent algorithms by proposing an online adaptation mechanism for coordinate frequencies, which removes the need for pre-estimation and automatically adjusts during optimization. The result is significant speed-ups over state-of-the-art training methods for various machine learning optimization problems.
Coordinate descent (CD) algorithms have become the method of choice for solving a number of optimization problems in machine learning. They are particularly popular for training linear models, including linear support vector machine classification, LASSO regression, and logistic regression. We consider general CD with non-uniform selection of coordinates. Instead of fixing selection frequencies beforehand we propose an online adaptation mechanism for this important parameter, called the adaptive coordinate frequencies (ACF) method. This mechanism removes the need to estimate optimal coordinate frequencies beforehand, and it automatically reacts to changing requirements during an optimization run. We demonstrate the usefulness of our ACF-CD approach for a variety of optimization problems arising in machine learning contexts. Our algorithm offers significant speed-ups over state-of-the-art training methods.