Stimulus symmetries can confound representational similarity analyses
For researchers using RSMs to study neural codes, this work reveals a previously unrecognized confound that can undermine the reliability of RSM-based analyses.
The paper shows that symmetries in network inputs can cause representational similarity matrices (RSMs) to vary even when representations are functionally equivalent, leading to drifting RSMs in networks trained with SGD or regularization. This is demonstrated in image-trained networks where the symmetry is latent.
What can representational similarity matrices (RSMs) tell us about a neural code? As the popularity of these summary statistics grows, so too does the need for a more complete characterization of their properties. Here, we show that symmetries in network inputs can confound RSM-based analyses. Stimulus symmetries render many representations functionally equivalent, but these different configurations can lead to different RSMs. These different RSMs reflect qualitatively different representational geometries. We show that stochastic gradient descent or energetic regularization can generate sparse, drifting codes, leading in turn to drifting RSMs. Moreover, we demonstrate that these phenomena are present in networks trained to encode image data, where the symmetry is latent. Our results illustrate the challenges inherent in comparing nonlinear neural codes, when functionally-equivalent representations are not related by a simple rotation.