Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA
It provides a theoretical bridge between spectral and probabilistic approaches in dimensionality reduction, which is incremental in nature.
This work tackles the problem of connecting different dimensionality reduction methods by showing a theoretical link between Locally Linear Embedding (LLE), factor analysis, and probabilistic PCA, revealing that LLE's linear reconstruction step can be interpreted stochastically using expectation maximization.
Locally Linear Embedding (LLE) is a nonlinear spectral dimensionality reduction and manifold learning method. It has two main steps which are linear reconstruction and linear embedding of points in the input space and embedding space, respectively. In this work, we look at the linear reconstruction step from a stochastic perspective where it is assumed that every data point is conditioned on its linear reconstruction weights as latent factors. The stochastic linear reconstruction of LLE is solved using expectation maximization. We show that there is a theoretical connection between three fundamental dimensionality reduction methods, i.e., LLE, factor analysis, and probabilistic Principal Component Analysis (PCA). The stochastic linear reconstruction of LLE is formulated similar to the factor analysis and probabilistic PCA. It is also explained why factor analysis and probabilistic PCA are linear and LLE is a nonlinear method. This work combines and makes a bridge between two broad approaches of dimensionality reduction, i.e., the spectral and probabilistic algorithms.