Context-Free Transductions with Neural Stacks
This addresses the challenge of integrating structured memory into neural networks for sequence processing, but it is incremental as it builds on existing stack RNN models.
The paper investigates stack-augmented RNNs for tasks like string reversal and context-free language modeling, finding they can learn stack-based strategies but are harder to train than LSTMs and sometimes use the stack as unstructured memory.
This paper analyzes the behavior of stack-augmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, context-free language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.