Coordinate Descent with Arbitrary Sampling II: Expected Separable Overapproximation
This work addresses a bottleneck in designing and analyzing coordinate descent methods for optimization, but it is incremental as it extends prior results from uniform to arbitrary samplings.
The paper tackles the problem of deriving expected separable overapproximation (ESO) inequalities for randomized coordinate descent methods with arbitrary samplings, and it develops a systematic technique that recovers existing results for a large class of functions.
The design and complexity analysis of randomized coordinate descent methods, and in particular of variants which update a random subset (sampling) of coordinates in each iteration, depends on the notion of expected separable overapproximation (ESO). This refers to an inequality involving the objective function and the sampling, capturing in a compact way certain smoothness properties of the function in a random subspace spanned by the sampled coordinates. ESO inequalities were previously established for special classes of samplings only, almost invariably for uniform samplings. In this paper we develop a systematic technique for deriving these inequalities for a large class of functions and for arbitrary samplings. We demonstrate that one can recover existing ESO results using our general approach, which is based on the study of eigenvalues associated with samplings and the data describing the function.