Infinite-dimensional reservoir computing
This provides a theoretical foundation for reservoir computing algorithms in dynamic contexts, though it appears incremental as it extends existing generalized Barron functionals.
The paper tackles the problem of approximating and generalizing input/output systems in reservoir computing by introducing a new concept class of infinite-dimensional state-space systems, proving that randomly generated echo state networks with linear or ReLU activations can achieve this with provable convergence guarantees free from the curse of dimensionality.
Reservoir computing approximation and generalization bounds are proved for a new concept class of input/output systems that extends the so-called generalized Barron functionals to a dynamic context. This new class is characterized by the readouts with a certain integral representation built on infinite-dimensional state-space systems. It is shown that this class is very rich and possesses useful features and universal approximation properties. The reservoir architectures used for the approximation and estimation of elements in the new class are randomly generated echo state networks with either linear or ReLU activation functions. Their readouts are built using randomly generated neural networks in which only the output layer is trained (extreme learning machines or random feature neural networks). The results in the paper yield a fully implementable recurrent neural network-based learning algorithm with provable convergence guarantees that do not suffer from the curse of dimensionality.