The Effect of Masking Strategies on Knowledge Retention by Language Models
This work addresses the problem of knowledge loss in language models for researchers and practitioners in NLP, though it is incremental as it builds on existing pre-training methods.
The paper investigates how different masking strategies during pre-training affect language models' retention of factual knowledge, finding that masking entities and correlated spans based on pointwise mutual information leads to better knowledge retention compared to random token masking, with results showing reduced catastrophic forgetting.
Language models retain a significant amount of world knowledge from their pre-training stage. This allows knowledgeable models to be applied to knowledge-intensive tasks prevalent in information retrieval, such as ranking or question answering. Understanding how and which factual information is acquired by our models is necessary to build responsible models. However, limited work has been done to understand the effect of pre-training tasks on the amount of knowledge captured and forgotten by language models during pre-training. Building a better understanding of knowledge acquisition is the goal of this paper. Therefore, we utilize a selection of pre-training tasks to infuse knowledge into our model. In the following steps, we test the model's knowledge retention by measuring its ability to answer factual questions. Our experiments show that masking entities and principled masking of correlated spans based on pointwise mutual information lead to more factual knowledge being retained than masking random tokens. Our findings demonstrate that, like the ability to perform a task, the (factual) knowledge acquired from being trained on that task is forgotten when a model is trained to perform another task (catastrophic forgetting) and how to prevent this phenomenon. To foster reproducibility, the code, as well as the data used in this paper, are openly available.