Pipelined Decoder for Efficient Context-Aware Text Generation
This addresses a key efficiency problem for users of generative AI models, offering a practical speed-up for context-aware text generation tasks, though it is incremental as it builds on existing decoder architectures.
The paper tackles the bottleneck of slow token-by-token generation in autoregressive models by proposing a pipelined decoder that generates multiple subsequences in parallel, significantly improving generation speed without major quality loss or extra memory use across tasks like question answering and summarization.
As the basis of generative AI, an autoregressive model requires the generation of a new token depending on all the previously generated tokens, which brings high quality but also restricts the model to generate tokens one by one, forming a bottleneck limiting the generation speed. In this paper, we propose a new decoder architecture that efficiently generates text in parallel for context-aware generation tasks. Our proposed pipelined decoder initiates the generation of multiple subsequences simultaneously, and, at each time-step, it generates a new token for each subsequence to realize parallelism. Experiments on multiple text generation tasks, including question answering, text summarization, and keyphrase generation, show that our pipelined decoder significantly improves the generation speed without a significant loss of generation quality or additional memory consumption.