Scalable and Interpretable Representation Alignment with Ordinal Similarity
For researchers in representation learning, this provides a principled and practical metric to evaluate and understand representations, addressing key limitations of existing metrics.
The paper proposes ordinal-similarity indices (TSI and QSI) for measuring representation alignment, which are interpretable, robust to outliers, and computationally efficient, enabling scalable evaluation without heuristic approximations.
Evaluating representation similarity is fundamental to representation learning. However, existing metrics suffer from significant limitations: they lack interpretability due to shifting baselines, lack robustness to outliers, and are computationally intractable for large datasets, forcing reliance on heuristic approximations. To address this, we develop an ordinal-similarity framework, instantiated by the Triplet (TSI) and Quadruplet (QSI) Similarity Indices, which measure alignment by quantifying the consistency of ordinal relationships. We theoretically demonstrate this formulation is inherently interpretable, robust to outliers, and computationally efficient. Finally, we establish a formal equivalence between TSI and local neighborhood alignment, measured by Mutual Nearest Neighbors. Empirically, we validate these properties and show that ordinal similarity offers a scalable approach to measuring alignment, enabling practitioners to better understand and design representations.