Unsupervised Improvement of Factual Knowledge in Language Models
This addresses the issue of improving factual knowledge in language models for applications requiring accurate information retrieval, though it appears incremental as it builds on existing MLM pretraining.
The paper tackled the problem of masked language modeling (MLM) being dominated by high-frequency words, which is sub-optimal for learning factual knowledge, and proposed an unsupervised approach to prioritize informative words, resulting in significant performance improvements on knowledge-intensive tasks like factual recall and question answering.
Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we propose an approach for influencing MLM pretraining in a way that can improve language model performance on a variety of knowledge-intensive tasks. We force the language model to prioritize informative words in a fully unsupervised way. Experiments demonstrate that the proposed approach can significantly improve the performance of pretrained language models on tasks such as factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting.