Efficient Cross-Validation of Echo State Networks
This is an incremental improvement for practitioners using ESNs in time series analysis, offering more reliable validation without significant computational overhead.
The paper tackles the problem of hyper-parameter tuning in Echo State Networks (ESNs) by proposing efficient cross-validation schemes that maintain constant time complexity for the dominant training component, making k-fold validation nearly as fast as single-split validation in many cases.
Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cross-validating ESNs and introduce an efficient algorithm for implementing them. The component that dominates the time complexity of the already quite fast ESN training remains constant (does not scale up with $k$) in our proposed method of doing $k$-fold cross-validation. The component that does scale linearly with $k$ starts dominating only in some not very common situations. Thus in many situations $k$-fold cross-validation of ESNs can be done for virtually the same time complexity as a simple single split validation. Space complexity can also remain the same. We also discuss when the proposed validation schemes for ESNs could be beneficial and empirically investigate them on several different real-world datasets.