Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
This addresses the hyper-parameter tuning problem for researchers and practitioners using ESNs, offering a more stable and reliable alternative, though it appears incremental as it builds on existing ESN frameworks.
The paper tackles the sensitivity of Echo State Networks (ESNs) to hyper-parameters by proposing a model that eliminates critical dependence, resulting in networks that provably avoid chaos and maintain high memory of past inputs comparable to linear networks, with experiments showing it approximates common nonlinear benchmarks.
Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory--nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behaviour in phase space characterised by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking.