MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
This work addresses the limitations of existing Chinese BERT vocabularies and single-vocabulary pretraining for Chinese NLP researchers and practitioners, offering an incremental improvement.
The authors propose a new vocabulary construction method, seg_tok, for Chinese BERT models, which combines Chinese word segmentation and subword tokenization. They also introduce three multi-vocabulary pretraining (MVP) methods to enhance model expressiveness. Experiments show that seg_tok improves performance on sentence-level tasks and efficiency compared to character-based vocabularies, and MVP further boosts downstream performance, particularly for seg_tok on sequence labeling tasks.
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert \cite{devlin2018bert}, which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, \emph{seg\_tok}, to form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models expressiveness. Experiments show that: (a) compared with char based vocabulary, \emph{seg\_tok} does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve \emph{seg\_tok}'s performances on sequence labeling tasks.