Learning Reservoir Dynamics with Temporal Self-Modulation
This work addresses a performance bottleneck in reservoir computing for time-series processing, offering a physically implementable improvement for edge AI applications, though it appears incremental as it builds directly on existing RC methods.
The paper tackled the problem of reservoir computing (RC) having inferior learning performance compared to state-of-the-art RNN models by proposing self-modulated RC (SM-RC) with a self-modulation mechanism using input and reservoir gates, resulting in SM-RC outperforming RC in tasks like NARMA and Lorentz models, achieving higher prediction accuracy with a reservoir 10 times larger in Lorentz tasks.
Reservoir computing (RC) can efficiently process time-series data by transferring the input signal to randomly connected recurrent neural networks (RNNs), which are referred to as a reservoir. The high-dimensional representation of time-series data in the reservoir significantly simplifies subsequent learning tasks. Although this simple architecture allows fast learning and facile physical implementation, the learning performance is inferior to that of other state-of-the-art RNN models. In this paper, to improve the learning ability of RC, we propose self-modulated RC (SM-RC), which extends RC by adding a self-modulation mechanism. The self-modulation mechanism is realized with two gating variables: an input gate and a reservoir gate. The input gate modulates the input signal, and the reservoir gate modulates the dynamical properties of the reservoir. We demonstrated that SM-RC can perform attention tasks where input information is retained or discarded depending on the input signal. We also found that a chaotic state emerged as a result of learning in SM-RC. This indicates that self-modulation mechanisms provide RC with qualitatively different information-processing capabilities. Furthermore, SM-RC outperformed RC in NARMA and Lorentz model tasks. In particular, SM-RC achieved a higher prediction accuracy than RC with a reservoir 10 times larger in the Lorentz model tasks. Because the SM-RC architecture only requires two additional gates, it is physically implementable as RC, providing a new direction for realizing edge AI.