On Nonlinear Dimensionality Reduction, Linear Smoothing and Autoencoding
This work provides a foundational theory for NLDR, addressing a gap in understanding for researchers in machine learning and statistics, though it is incremental in building on existing methods.
The paper tackles the limited theoretical understanding of nonlinear dimensionality reduction (NLDR) methods by developing a novel framework that connects NLDR to linear smoothing theory, enabling analysis of existing methods and derivation of new ones, with results showing that Local Tangent Space Alignment is superior within a specific class of methods.
We develop theory for nonlinear dimensionality reduction (NLDR). A number of NLDR methods have been developed, but there is limited understanding of how these methods work and the relationships between them. There is limited basis for using existing NLDR theory for deriving new algorithms. We provide a novel framework for analysis of NLDR via a connection to the statistical theory of linear smoothers. This allows us to both understand existing methods and derive new ones. We use this connection to smoothing to show that asymptotically, existing NLDR methods correspond to discrete approximations of the solutions of sets of differential equations given a boundary condition. In particular, we can characterize many existing methods in terms of just three limiting differential operators and boundary conditions. Our theory also provides a way to assert that one method is preferable to another; indeed, we show Local Tangent Space Alignment is superior within a class of methods that assume a global coordinate chart defines an isometric embedding of the manifold.