Navigating the Effect of Parametrization for Dimensionality Reduction
This addresses a problem for practitioners in machine learning by clarifying performance differences and offering an improved parametric method, though it is incremental as it builds on existing parametric approaches.
The paper tackles the misconception that parametric and non-parametric dimensionality reduction methods perform equivalently, showing that parametric methods lose local details, and introduces ParamRepulsor, which achieves state-of-the-art performance in local structure preservation for parametric methods without compromising global structure.
Parametric dimensionality reduction methods have gained prominence for their ability to generalize to unseen datasets, an advantage that traditional approaches typically lack. Despite their growing popularity, there remains a prevalent misconception among practitioners about the equivalence in performance between parametric and non-parametric methods. Here, we show that these methods are not equivalent -- parametric methods retain global structure but lose significant local details. To explain this, we provide evidence that parameterized approaches lack the ability to repulse negative pairs, and the choice of loss function also has an impact. Addressing these issues, we developed a new parametric method, ParamRepulsor, that incorporates Hard Negative Mining and a loss function that applies a strong repulsive force. This new method achieves state-of-the-art performance on local structure preservation for parametric methods without sacrificing the fidelity of global structural representation. Our code is available at https://github.com/hyhuang00/ParamRepulsor.