Optimal Sketching Bounds for Exp-concave Stochastic Minimization
This work addresses optimization efficiency for exp-concave problems, connecting algorithmic stability to sketching methods, but appears incremental in its technical contributions.
The authors derived optimal statistical and computational complexity bounds for exp-concave stochastic minimization using effective dimension, which is smaller than ambient dimension for common covariance eigendecays, and introduced a fast sketch-to-precondition implementation.
We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension. For common eigendecay patterns of the population covariance matrix, this quantity is significantly smaller than the ambient dimension. Our results reveal interesting connections to sketching results in numerical linear algebra. In particular, our statistical analysis highlights a novel and natural relationship between algorithmic stability of empirical risk minimization and ridge leverage scores, which play significant role in sketching-based methods. Our main computational result is a fast implementation of a sketch-to-precondition approach in the context of exp-concave empirical risk minimization.