Exact Hard Monotonic Attention for Character-Level Transduction
This addresses the problem of improving accuracy and efficiency in character-level string-to-string transduction tasks, such as grapheme-to-phoneme conversion and morphological inflection, for natural language processing applications, though it is incremental as it builds on existing monotonic attention methods.
The paper tackles the question of whether monotonicity is a helpful inductive bias for character-level transduction tasks like morphological inflection, by developing a hard attention sequence-to-sequence model that enforces strict monotonicity and jointly learns latent alignments. The result is state-of-the-art performance on morphological inflection and strong performance on two other tasks, with exact marginalization over alignments achieved via dynamic programming.
Many common character-level, string-to string transduction tasks, e.g., grapheme-tophoneme conversion and morphological inflection, consist almost exclusively of monotonic transductions. However, neural sequence-to sequence models that use non-monotonic soft attention often outperform popular monotonic models. In this work, we ask the following question: Is monotonicity really a helpful inductive bias for these tasks? We develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns a latent alignment jointly while learning to transduce. With the help of dynamic programming, we are able to compute the exact marginalization over all monotonic alignments. Our models achieve state-of-the-art performance on morphological inflection. Furthermore, we find strong performance on two other character-level transduction tasks. Code is available at https://github.com/shijie-wu/neural-transducer.