Learning Better Masking for Better Language Model Pre-training
This work addresses a specific bottleneck in language model pre-training for NLP researchers, offering incremental improvements over existing methods.
The paper tackles the suboptimal performance of fixed masking strategies in Masked Language Modeling by proposing two scheduled masking approaches that adaptively adjust masking ratio and content during training, resulting in improved pre-training efficiency and effectiveness on downstream tasks.
Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre-training.