Style Locality for Controllable Generation with kNN Language Models
This work addresses the need for more precise style control in text generation for applications like content moderation or personalized communication, representing an incremental advance in memory-augmented models.
The paper tackled the problem of controllable text generation by incorporating locality levels into nearest neighbor language models, resulting in improved control over style attributes like politeness and formality with a better fluency-style trade-off than prior methods.
Recent language models have been improved by the addition of external memory. Nearest neighbor language models retrieve similar contexts to assist in word prediction. The addition of locality levels allows a model to learn how to weight neighbors based on their relative location to the current text in source documents, and have been shown to further improve model performance. Nearest neighbor models have been explored for controllable generation but have not examined the use of locality levels. We present a novel approach for this purpose and evaluate it using automatic and human evaluation on politeness, formality, supportiveness, and toxicity textual data. We find that our model is successfully able to control style and provides a better fluency-style trade-off than previous work.