Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
This work provides insights into the representational capacity and learning dynamics of RNNs for signal integration, which is relevant for researchers studying neural network interpretability and computational neuroscience.
This paper investigates Recurrent Neural Networks (RNNs) trained to integrate one or multiple temporal signals, characterizing the conditions under which an RNN with 'n' neurons can integrate 'D(n)' scalar signals of arbitrary duration. It demonstrates that the internal state of these networks, for both linear and ReLU neurons, resides close to a D-dimensional manifold, with each neuron contributing to the information of all integrals.
We study the learning dynamics and the representations emerging in Recurrent Neural Networks trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which a RNN with n neurons learns to integrate D(n) scalar signals of arbitrary duration. We show, both for linear and ReLU neurons, that its internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.