A Random Matrix Perspective of Echo State Networks: From Precise Bias--Variance Characterization to Optimal Regularization
This work offers interpretable theory and practical tuning guidelines for ESNs, addressing a domain-specific problem in reservoir computing, but it is incremental as it builds on existing teacher-student frameworks.
The paper tackled the theoretical analysis of Echo State Networks (ESNs) by deriving closed-form expressions for bias, variance, and mean-squared error using random matrix theory, revealing that ESNs avoid double descent and achieve lower MSE with limited data and memory, and provided optimal regularization formulas.
We present a rigorous asymptotic analysis of Echo State Networks (ESNs) in a teacher student setting with a linear teacher with oracle weights. Leveraging random matrix theory, we derive closed form expressions for the asymptotic bias, variance, and mean-squared error (MSE) as functions of the input statistics, the oracle vector, and the ridge regularization parameter. The analysis reveals two key departures from classical ridge regression: (i) ESNs do not exhibit double descent, and (ii) ESNs attain lower MSE when both the number of training samples and the teacher memory length are limited. We further provide an explicit formula for the optimal regularization in the identity input covariance case, and propose an efficient numerical scheme to compute the optimum in the general case. Together, these results offer interpretable theory and practical guidelines for tuning ESNs, helping reconcile recent empirical observations with provable performance guarantees