Iteratively-Reweighted Least-Squares Fitting of Support Vector Machines: A Majorization--Minimization Algorithm Approach
This work addresses the computational challenge of SVM fitting for data analysts by offering a new algorithmic framework, though it appears incremental as it builds on existing MM methods rather than introducing a paradigm shift.
The authors tackled the problem of fitting support vector machines (SVMs) by proposing an alternative approach using the majorization-minimization (MM) paradigm to construct iteratively-reweighted least-squares (IRLS) algorithms, resulting in algorithms that monotonically decrease objectives and are globally convergent to stationary points for various loss functions and penalizations.
Support vector machines (SVMs) are an important tool in modern data analysis. Traditionally, support vector machines have been fitted via quadratic programming, either using purpose-built or off-the-shelf algorithms. We present an alternative approach to SVM fitting via the majorization--minimization (MM) paradigm. Algorithms that are derived via MM algorithm constructions can be shown to monotonically decrease their objectives at each iteration, as well as be globally convergent to stationary points. We demonstrate the construction of iteratively-reweighted least-squares (IRLS) algorithms, via the MM paradigm, for SVM risk minimization problems involving the hinge, least-square, squared-hinge, and logistic losses, and 1-norm, 2-norm, and elastic net penalizations. Successful implementations of our algorithms are presented via some numerical examples.