Regularizers versus Losses for Nonlinear Dimensionality Reduction: A Factored View with New Convex Relaxations
This provides a foundational framework for understanding and improving dimensionality reduction techniques, though it is incremental in building on existing methods.
The paper tackles the problem of unifying non-parametric dimensionality reduction methods by showing they can be expressed as regularized loss minimization with singular value truncation, and it introduces new convex regularizers that combine distance maximization with rank reduction, applicable to any loss.
We demonstrate that almost all non-parametric dimensionality reduction methods can be expressed by a simple procedure: regularized loss minimization plus singular value truncation. By distinguishing the role of the loss and regularizer in such a process, we recover a factored perspective that reveals some gaps in the current literature. Beyond identifying a useful new loss for manifold unfolding, a key contribution is to derive new convex regularizers that combine distance maximization with rank reduction. These regularizers can be applied to any loss.