A Taxonomy for Neural Memory Networks
This work provides a conceptual framework for researchers and practitioners to select or design appropriate memory networks for specific tasks, but it is incremental as it organizes existing models rather than introducing new ones.
The authors proposed a taxonomy for neural memory networks based on memory organization, categorizing popular models like RNNs, LSTMs, neural stacks, and neural Turing machines, and analyzed their relative expressive power and task suitability.
In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our arguments. And we use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem.