The Ordered Weighted $\ell_1$ Norm: Atomic Formulation, Projections, and Algorithms
This work provides incremental improvements in optimization methods for a specific regularization technique in machine learning, benefiting researchers in sparse modeling and variable clustering.
The paper tackled the problem of efficiently solving linear regression with ordered weighted ℓ1 (OWL) norm regularization, which promotes sparsity and variable clustering, by deriving new formulations and algorithms, resulting in accelerated projected gradient methods being shown to be considerably faster than conditional gradient approaches in experiments.
The ordered weighted $\ell_1$ norm (OWL) was recently proposed, with two different motivations: its good statistical properties as a sparsity promoting regularizer; the fact that it generalizes the so-called {\it octagonal shrinkage and clustering algorithm for regression} (OSCAR), which has the ability to cluster/group regression variables that are highly correlated. This paper contains several contributions to the study and application of OWL regularization: the derivation of the atomic formulation of the OWL norm; the derivation of the dual of the OWL norm, based on its atomic formulation; a new and simpler derivation of the proximity operator of the OWL norm; an efficient scheme to compute the Euclidean projection onto an OWL ball; the instantiation of the conditional gradient (CG, also known as Frank-Wolfe) algorithm for linear regression problems under OWL regularization; the instantiation of accelerated projected gradient algorithms for the same class of problems. Finally, a set of experiments give evidence that accelerated projected gradient algorithms are considerably faster than CG, for the class of problems considered.