The Case for a Single Model that can Both Generate Continuations and Fill in the Blank
This addresses the need for more versatile natural language generation systems for writers, though it is incremental as it builds on existing pre-training methods.
The paper tackles the problem of creating a single model capable of both generating continuations and filling in blanks in text, showing that models pre-trained with a fill-in-the-blank objective can perform both tasks effectively, while continuation-only models cannot.
The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform _both_ FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.