Memory and forecasting capacities of nonlinear recurrent networks
This work provides theoretical foundations for analyzing memory and forecasting in recurrent networks, addressing a gap for dependent inputs, but it is incremental as it builds on existing capacity concepts.
The authors generalized memory capacity to nonlinear recurrent networks with dependent inputs and introduced forecasting capacity, deriving generic bounds based on network size and input statistics. They also proved that for linear networks with independent inputs, memory capacity equals the rank of the controllability matrix, confirming a long-assumed result.
The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs makes natural the introduction of the network forecasting capacity, that measures the possibility of forecasting time series values using network states. Generic bounds for memory and forecasting capacities are formulated in terms of the number of neurons of the nonlinear recurrent network and the autocovariance function or the spectral density of the input. These bounds generalize well-known estimates in the literature to a dependent inputs setup. Finally, for the particular case of linear recurrent networks with independent inputs it is proved that the memory capacity is given by the rank of the associated controllability matrix, a fact that has been for a long time assumed to be true without proof by the community.