Reservoir Memory Machines as Neural Computers
This work addresses the problem of inefficient training for neural computers, offering a more practical solution for researchers and practitioners in machine learning, though it is incremental as it builds on existing echo state network methods.
The paper tackles the challenge of training differentiable neural computers by proposing an echo state network with explicit memory, which can be trained efficiently and recognizes all regular languages, performing comparably to fully-trained deep models on benchmark tasks.
Differentiable neural computers extend artificial neural networks with an explicit memory without interference, thus enabling the model to perform classic computation tasks such as graph traversal. However, such models are difficult to train, requiring long training times and large datasets. In this work, we achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently, namely an echo state network with an explicit memory without interference. This extension enables echo state networks to recognize all regular languages, including those that contractive echo state networks provably can not recognize. Further, we demonstrate experimentally that our model performs comparably to its fully-trained deep version on several typical benchmark tasks for differentiable neural computers.