InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model
This work addresses efficiency and flexibility issues in insertion-based text generation for NLP researchers and practitioners, offering incremental improvements over existing methods.
The authors tackled the inefficiency and inflexibility of insertion-based text generation models by proposing InsNet, which uses novel position encoding and slot representation to enable one-pass context encoding during training and a flexible decoding algorithm. Experiments on text generation and machine translation datasets showed improvements in training speed, inference efficiency, and generation quality over previous methods.
We propose InsNet, an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential). Unlike most existing insertion-based text generation works that require re-encoding of the context after each insertion operation and thus are inefficient to train, InsNet only requires one pass of context encoding for the entire sequence during training by introducing a novel insertion-oriented position encoding and a light-weighted slot representation strategy to enable computation sharing. Furthermore, we propose an algorithm InsNet-Dinic to better determine the parallelization of insertion operations that provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such as machine translation with efficient decoding, or less parallelizable tasks such as open-domain text generation to guarantee high-quality outputs. Experiments on two lexically constrained text generation datasets and three machine translation datasets demonstrate InsNet's advantages over previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.