Encode, Tag, Realize: High-Precision Text Editing
This addresses the problem of efficient and high-precision text generation for applications like summarization and grammar correction, though it is incremental as it builds on existing tagging and Transformer methods.
The paper tackles text generation by framing it as a text editing task using a sequence tagging approach called LaserTagger, which achieves new state-of-the-art results on three out of four English text tasks and is over two orders of magnitude faster at inference than comparable models.
We propose LaserTagger - a sequence tagging approach that casts text generation as a text editing task. Target texts are reconstructed from the inputs using three main edit operations: keeping a token, deleting it, and adding a phrase before the token. To predict the edit operations, we propose a novel model, which combines a BERT encoder with an autoregressive Transformer decoder. This approach is evaluated on English text on four tasks: sentence fusion, sentence splitting, abstractive summarization, and grammar correction. LaserTagger achieves new state-of-the-art results on three of these tasks, performs comparably to a set of strong seq2seq baselines with a large number of training examples, and outperforms them when the number of examples is limited. Furthermore, we show that at inference time tagging can be more than two orders of magnitude faster than comparable seq2seq models, making it more attractive for running in a live environment.