Learning Transductions and Alignments with RNN Seq2seq Models
This work addresses the generalization limitations of RNN seq2seq models for formal transduction tasks, which is an incremental insight into their capabilities compared to finite state transducers.
The paper investigates the ability of RNN seq2seq models to learn four transduction tasks (identity, reversal, total reduplication, and quadratic copying), finding that they only approximate mappings from training data and fail to generalize out-of-distribution, with attention improving efficiency but not overcoming this limitation.
The paper studies the capabilities of Recurrent-Neural-Network sequence to sequence (RNN seq2seq) models in learning four transduction tasks: identity, reversal, total reduplication, and quadratic copying. These transductions are traditionally well studied under finite state transducers and attributed with increasing complexity. We find that RNN seq2seq models are only able to approximate a mapping that fits the training or in-distribution data, instead of learning the underlying functions. Although attention makes learning more efficient and robust, it does not overcome the out-of-distribution generalization limitation. We establish a novel complexity hierarchy for learning the four tasks for attention-less RNN seq2seq models, which may be understood in terms of the complexity hierarchy of formal languages, instead of string transductions. RNN variants also play a role in the results. In particular, we show that Simple RNN seq2seq models cannot count the input length.