Learning to Plan for Language Modeling from Unlabeled Data
This addresses the problem of planning in language modeling for applications requiring structured text generation, but it is incremental as it extends existing language model frameworks with an external planner.
The paper tackles the limitation of next-token-prediction in language models for planning tasks like coherent writing by introducing a self-supervised planning module that predicts future abstract writing actions, improving language modeling performance, particularly in text structure.
By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns to predict future abstract writing actions, which correspond to centroids in a clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.