Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
This work provides a systematic guide for choosing similarity metrics in AI and neuroscience, though it is incremental as it compares existing metrics rather than introducing new ones.
The paper tackled the lack of systematic comparisons of representational similarity metrics by introducing a quantitative framework to evaluate their discriminative power across model families and training regimes, showing that separability increases with stricter alignment constraints and identifying soft-matching as the top performer.
Representational similarity metrics are fundamental tools in neuroscience and AI, yet we lack systematic comparisons of their discriminative power across model families. We introduce a quantitative framework to evaluate representational similarity measures based on their ability to separate model families-across architectures (CNNs, Vision Transformers, Swin Transformers, ConvNeXt) and training regimes (supervised vs. self-supervised). Using three complementary separability measures-dprime from signal detection theory, silhouette coefficients and ROC-AUC, we systematically assess the discriminative capacity of commonly used metrics including RSA, linear predictivity, Procrustes, and soft matching. We show that separability systematically increases as metrics impose more stringent alignment constraints. Among mapping-based approaches, soft-matching achieves the highest separability, followed by Procrustes alignment and linear predictivity. Non-fitting methods such as RSA also yield strong separability across families. These results provide the first systematic comparison of similarity metrics through a separability lens, clarifying their relative sensitivity and guiding metric choice for large-scale model and brain comparisons.