Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?
This is an incremental result that questions the utility of a common metric in machine learning for researchers and practitioners.
The paper tackled the problem of the memory capacity metric's usefulness for distinguishing recurrent neural network performance, showing that for nonlinear RNNs, memory capacity yields arbitrary values within bounds based on input scale, confirming it has no practical value.
The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This work shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending exclusively on the scale of the input process. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.