The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations
This provides a more holistic tool for model selection and analysis in scientific research, though it is incremental as it builds on existing methods like CKA and Procrustes.
The authors tackled the problem of comparing neural network representations by proposing the Triangle of Similarity framework, which combines static, functional, and sparsity perspectives, and found that architectural family determines similarity clusters, CKA self-similarity correlates with task accuracy during pruning, and pruning can regularize representations to expose shared computational cores.
Comparing neural network representations is essential for understanding and validating models in scientific applications. Existing methods, however, often provide a limited view. We propose the Triangle of Similarity, a framework that combines three complementary perspectives: static representational similarity (CKA/Procrustes), functional similarity (Linear Mode Connectivity or Predictive Similarity), and sparsity similarity (robustness under pruning). Analyzing a range of CNNs, Vision Transformers, and Vision-Language Models using both in-distribution (ImageNetV2) and out-of-distribution (CIFAR-10) testbeds, our initial findings suggest that: (1) architectural family is a primary determinant of representational similarity, forming distinct clusters; (2) CKA self-similarity and task accuracy are strongly correlated during pruning, though accuracy often degrades more sharply; and (3) for some model pairs, pruning appears to regularize representations, exposing a shared computational core. This framework offers a more holistic approach for assessing whether models have converged on similar internal mechanisms, providing a useful tool for model selection and analysis in scientific research.