Separation of Memory and Processing in Dual Recurrent Neural Networks
This work addresses the challenge of making neural networks more interpretable and efficient for specific computational tasks, though it appears incremental as it builds on existing architectures like Elman and LSTM.
The authors tackled the problem of improving interpretability and accuracy in recurrent neural networks by introducing noise to force binary activations, resulting in models that behave like finite automata and achieve higher accuracy on tasks such as regular language recognition and arithmetic computation.
We explore a neural network architecture that stacks a recurrent layer and a feedforward layer that is also connected to the input, and compare it to standard Elman and LSTM architectures in terms of accuracy and interpretability. When noise is introduced into the activation function of the recurrent units, these neurons are forced into a binary activation regime that makes the networks behave much as finite automata. The resulting models are simpler, easier to interpret and get higher accuracy on different sample problems, including the recognition of regular languages, the computation of additions in different bases and the generation of arithmetic expressions.