Compressive Statistical Learning with Random Feature Moments
This addresses the challenge of scaling learning algorithms for large datasets, though it appears incremental as it builds on existing compressive and statistical learning concepts.
The authors tackled the problem of resource-efficient large-scale learning by compressing training data into a low-dimensional sketch of random empirical moments, enabling near-minimizer computation through nonlinear least squares, with results including sufficient sketch sizes to control generalization error for tasks like PCA, clustering, and Gaussian mixture modeling.
We describe a general framework -- compressive statistical learning -- for resource-efficient large-scale learning: the training collection is compressed in one pass into a low-dimensional sketch (a vector of random empirical generalized moments) that captures the information relevant to the considered learning task. A near-minimizer of the risk is computed from the sketch through the solution of a nonlinear least squares problem. We investigate sufficient sketch sizes to control the generalization error of this procedure. The framework is illustrated on compressive PCA, compressive clustering, and compressive Gaussian mixture Modeling with fixed known variance. The latter two are further developed in a companion paper.