Preserving Vector Space Properties in Dimensionality Reduction: A Relationship Preserving Loss Framework
This addresses a critical issue for tasks such as cross-modal retrieval, clustering, and classification, though it appears incremental as it builds on existing loss function frameworks.
The paper tackles the problem of dimensionality reduction distorting vector space properties like orthogonality and linear independence, proposing a Relationship Preserving Loss (RPL) that minimizes discrepancies between relationship matrices to preserve these properties, with initial experiments showing it reduces embedding dimensions while largely retaining downstream task performance.
Dimensionality reduction can distort vector space properties such as orthogonality and linear independence, which are critical for tasks including cross-modal retrieval, clustering, and classification. We propose a Relationship Preserving Loss (RPL), a loss function that preserves these properties by minimizing discrepancies between relationship matrices (e.g., Gram or cosine) of high-dimensional data and their low-dimensional embeddings. RPL trains neural networks for non-linear projections and is supported by error bounds derived from matrix perturbation theory. Initial experiments suggest that RPL reduces embedding dimensions while largely retaining performance on downstream tasks, likely due to its preservation of key vector space properties. While we describe here the use of RPL in dimensionality reduction, this loss can also be applied more broadly, for example to cross-domain alignment and transfer learning, knowledge distillation, fairness and invariance, dehubbing, graph and manifold learning, and federated learning, where distributed embeddings must remain geometrically consistent.