Fast and Simple PCA via Convex Optimization
This provides a faster method for PCA, a fundamental technique in machine learning and data analysis, but it appears incremental as it builds on recent advances in stochastic convex optimization.
The paper tackles the problem of principal component analysis (PCA) by reducing it to solving a small number of well-conditioned convex optimization problems, resulting in a new efficient method with time complexities of $ ilde{O}\left( rac{d}{\delta^2}+N ight)$ for computing a vector close to the top eigenvector and $ ilde{O}(d/\epsilon^2)$ for approximating the top eigenvalue, which are claimed to be the fastest to date for a wide parameter regime.
The problem of principle component analysis (PCA) is traditionally solved by spectral or algebraic methods. We show how computing the leading principal component could be reduced to solving a \textit{small} number of well-conditioned {\it convex} optimization problems. This gives rise to a new efficient method for PCA based on recent advances in stochastic methods for convex optimization. In particular we show that given a $d\times d$ matrix $\X = \frac{1}{n}\sum_{i=1}^n\x_i\x_i^{\top}$ with top eigenvector $\u$ and top eigenvalue $λ_1$ it is possible to: \begin{itemize} \item compute a unit vector $\w$ such that $(\w^{\top}\u)^2 \geq 1-ε$ in $\tilde{O}\left({\frac{d}{δ^2}+N}\right)$ time, where $δ= λ_1 - λ_2$ and $N$ is the total number of non-zero entries in $\x_1,...,\x_n$, \item compute a unit vector $\w$ such that $\w^{\top}\X\w \geq λ_1-ε$ in $\tilde{O}(d/ε^2)$ time. \end{itemize} To the best of our knowledge, these bounds are the fastest to date for a wide regime of parameters. These results could be further accelerated when $δ$ (in the first case) and $ε$ (in the second case) are smaller than $\sqrt{d/N}$.