Nearest Neighbor Non-autoregressive Text Generation
This addresses the trade-off between speed and quality in text generation for NLP applications, though it is incremental as it builds on existing iterative decoding methods.
The study tackled the quality degradation in non-autoregressive text generation by using nearest neighbors as initial states and editing them iteratively, achieving a 1.69-point improvement in translation quality with one-eighteenth fewer decoding iterations on a benchmark dataset.
Non-autoregressive (NAR) models can generate sentences with less computation than autoregressive models but sacrifice generation quality. Previous studies addressed this issue through iterative decoding. This study proposes using nearest neighbors as the initial state of an NAR decoder and editing them iteratively. We present a novel training strategy to learn the edit operations on neighbors to improve NAR text generation. Experimental results show that the proposed method (NeighborEdit) achieves higher translation quality (1.69 points higher than the vanilla Transformer) with fewer decoding iterations (one-eighteenth fewer iterations) on the JRC-Acquis En-De dataset, the common benchmark dataset for machine translation using nearest neighbors. We also confirm the effectiveness of the proposed method on a data-to-text task (WikiBio). In addition, the proposed method outperforms an NAR baseline on the WMT'14 En-De dataset. We also report analysis on neighbor examples used in the proposed method.