Blank Language Models
This addresses text manipulation challenges for applications such as style transfer and ancient text restoration, though it appears incremental as it builds on existing language model frameworks.
The authors tackled the problem of text editing and rewriting by proposing the Blank Language Model (BLM), which dynamically creates and fills blanks in sequences, resulting in significant outperformance over baselines in accuracy and fluency for tasks like filling missing text snippets.
We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM significantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restoration demonstrate the potential of this framework for a wide range of applications.