Representation biases: will we achieve complete understanding by analyzing representations?
This work highlights a critical limitation in using representational analysis for understanding systems, particularly in neuroscience and machine learning, by showing that biases can distort interpretations, making it incremental in refining existing methodologies.
The paper examines how biases in learned feature representations, such as over-representing simple features and under-representing complex ones, can lead to misleading inferences from common analyses like PCA and RSA, challenging the goal of achieving complete system understanding through representational analysis.
A common approach in neuroscience is to study neural representations as a means to understand a system -- increasingly, by relating the neural representations to the internal representations learned by computational models. However, a recent work in machine learning (Lampinen, 2024) shows that learned feature representations may be biased to over-represent certain features, and represent others more weakly and less-consistently. For example, simple (linear) features may be more strongly and more consistently represented than complex (highly nonlinear) features. These biases could pose challenges for achieving full understanding of a system through representational analysis. In this perspective, we illustrate these challenges -- showing how feature representation biases can lead to strongly biased inferences from common analyses like PCA, regression, and RSA. We also present homomorphic encryption as a simple case study of the potential for strong dissociation between patterns of representation and computation. We discuss the implications of these results for representational comparisons between systems, and for neuroscience more generally.