Large-Scale Sparse Principal Component Analysis with Application to Text Data
This addresses the problem of scaling Sparse PCA for large datasets like text corpora, offering an interpretable alternative to topic models, though it is incremental as it builds on existing Sparse PCA methods.
The paper tackles the computational challenge of Sparse PCA by showing it can be easier than PCA in practice, enabling reliable application to large datasets through a feature elimination pre-processing result and a fast block coordinate ascent algorithm, with experiments on text corpora involving millions of documents and hundreds of thousands of features.
Sparse PCA provides a linear combination of small number of features that maximizes variance across data. Although Sparse PCA has apparent advantages compared to PCA, such as better interpretability, it is generally thought to be computationally much more expensive. In this paper, we demonstrate the surprising fact that sparse PCA can be easier than PCA in practice, and that it can be reliably applied to very large data sets. This comes from a rigorous feature elimination pre-processing result, coupled with the favorable fact that features in real-life data typically have exponentially decreasing variances, which allows for many features to be eliminated. We introduce a fast block coordinate ascent algorithm with much better computational complexity than the existing first-order ones. We provide experimental results obtained on text corpora involving millions of documents and hundreds of thousands of features. These results illustrate how Sparse PCA can help organize a large corpus of text data in a user-interpretable way, providing an attractive alternative approach to topic models.