Traveling Waves Encode the Recent Past and Enhance Sequence Learning
This work addresses a gap in neuroscience and machine learning by providing a computational model for traveling waves, potentially enhancing sequence learning for applications in AI and brain-inspired computing, though it is incremental as it builds on existing hypotheses.
The authors tackled the problem of understanding the computational role of traveling waves in neural activity by introducing a Wave-RNN model, which demonstrated faster learning and lower error in synthetic memory tasks and competitive performance in sequence modeling tasks compared to wave-free RNNs and complex gated architectures like LSTMs and GRUs.
Traveling waves of neural activity have been observed throughout the brain at a diversity of regions and scales; however, their precise computational role is still debated. One physically inspired hypothesis suggests that the cortical sheet may act like a wave-propagating system capable of invertibly storing a short-term memory of sequential stimuli through induced waves traveling across the cortical surface, and indeed many experimental results from neuroscience correlate wave activity with memory tasks. To date, however, the computational implications of this idea have remained hypothetical due to the lack of a simple recurrent neural network architecture capable of exhibiting such waves. In this work, we introduce a model to fill this gap, which we denote the Wave-RNN (wRNN), and demonstrate how such an architecture indeed efficiently encodes the recent past through a suite of synthetic memory tasks where wRNNs learn faster and reach significantly lower error than wave-free counterparts. We further explore the implications of this memory storage system on more complex sequence modeling tasks such as sequential image classification and find that wave-based models not only again outperform comparable wave-free RNNs while using significantly fewer parameters, but additionally perform comparably to more complex gated architectures such as LSTMs and GRUs.