Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks
This work provides a theoretical framework for understanding memory limits in recurrent neural networks, relevant to reservoir computing and neuromorphic systems, though the results are primarily analytical with limited immediate practical impact.
The authors derive an inequality bounding short-term memory in dynamical systems, interpretable as an uncertainty relation between memory and input-induced fluctuations, and identify a suboptimal 'harmonic memory' that achieves the bound. They analytically and numerically validate the inequality in reservoir systems with input noise, and discover a 'noise-induced memory' phenomenon under state-space regularization where the uncertainty relation breaks down.
We present an inequality that bounds the short-term memory capability of dynamical systems from below. It can be interpreted as an uncertainty relation between a measure of short-term memory and that of the size of state fluctuations induced by input signals. The lower bound can be achieved by a readout weight and thus represents a suboptimal memory called harmonic memory. We examine analytically and numerically the inequality in a number of reservoir systems subject to input noise. We illustrate cases in which equality is achieved exactly, equality holds asymptotically, and the inequality is strict. We also study the effect of a state-space regularization to elucidate the inequality in terms of the fluctuation structure of the state-space. We find that a certain strength of input noise induces extra memory under the regularization, and we refer to this phenomenon as noise-induced memory. We observe that the memory uncertainty relation does not hold in general for the regularized memory and harmonic memory. This fact is explained in terms of the mechanism of noise-induced memory.