Deep Reservoir Networks with Learned Hidden Reservoir Weights using Direct Feedback Alignment
This work addresses a bottleneck in deep reservoir computing for researchers in time series analysis, offering a biologically-inspired method to enable learning in non-differentiable layers, but it appears incremental as it builds on existing reservoir and feedback alignment concepts.
The paper tackles the problem of learning hierarchical representations in deep reservoir networks, which are hindered by non-differentiable layers, by introducing a novel architecture that uses Direct Feedback Alignment for training. It demonstrates effectiveness on two real-world multidimensional time series datasets, though no concrete performance numbers are provided in the abstract.
Deep Reservoir Computing has emerged as a new paradigm for deep learning, which is based around the reservoir computing principle of maintaining random pools of neurons combined with hierarchical deep learning. The reservoir paradigm reflects and respects the high degree of recurrence in biological brains, and the role that neuronal dynamics play in learning. However, one issue hampering deep reservoir network development is that one cannot backpropagate through the reservoir layers. Recent deep reservoir architectures do not learn hidden or hierarchical representations in the same manner as deep artificial neural networks, but rather concatenate all hidden reservoirs together to perform traditional regression. Here we present a novel Deep Reservoir Network for time series prediction and classification that learns through the non-differentiable hidden reservoir layers using a biologically-inspired backpropagation alternative called Direct Feedback Alignment, which resembles global dopamine signal broadcasting in the brain. We demonstrate its efficacy on two real world multidimensional time series datasets.