CMV-BERT: Contrastive multi-vocab pretraining of BERT
This work aims to improve the pretraining effectiveness of language models, which is a foundational problem for natural language processing researchers.
This paper introduces CMV-BERT, a language model pretraining method that combines contrastive learning with multiple vocabularies (fine-grained and coarse-grained) to create different views of a sentence. The authors demonstrate that both components are beneficial and improve pretrained language models on downstream tasks.
In this work, we represent CMV-BERT, which improves the pretraining of a language model via two ingredients: (a) contrastive learning, which is well studied in the area of computer vision; (b) multiple vocabularies, one of which is fine-grained and the other is coarse-grained. The two methods both provide different views of an original sentence, and both are shown to be beneficial. Downstream tasks demonstrate our proposed CMV-BERT are effective in improving the pretrained language models.