Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representations
This provides a more direct and efficient framework for interpreting neural representations in neuroscience and AI, applicable across domains like language and vision, though it is incremental as it builds on existing second-order isomorphism methods.
The authors tackled the challenge of interpreting neural representations by introducing Metric Learning Encoding Models (MLEMs), which match distances in theoretical feature spaces to neural spaces, resulting in better recovery of ground-truth features in simulations and stronger robustness to noise in real language data compared to state-of-the-art methods like FR-RSA.
Understanding how explicit theoretical features are encoded in opaque neural systems is a central challenge now common to neuroscience and AI. We introduce Metric Learning Encoding Models (MLEMs) to address this challenge most directly as a metric learning problem: we fit the distance in the space of theoretical features to match the distance in neural space. Our framework improves on univariate encoding and decoding methods by building on second-order isomorphism methods, such as Representational Similarity Analysis, and extends them by learning a metric that efficiently models feature as well as interactions between them. The effectiveness of MLEM is validated through two sets of simulations. First, MLEMs recover ground-truth importance features in synthetic datasets better than state-of-the-art methods, such as Feature Reweighted RSA (FR-RSA). Second, we deploy MLEMs on real language data, where they show stronger robustness to noise in calculating the importance of linguistic features (gender, tense, etc.). MLEMs are applicable to any domains where theoretical features can be identified, such as language, vision, audition, etc. We release optimized code applicable to measure feature importance in the representations of any artificial neural networks or empirical neural data at https://github.com/LouisJalouzot/MLEM.