Pre-training with Meta Learning for Chinese Word Segmentation
This work addresses the discrepancy between pre-training and downstream tasks for Chinese Word Segmentation, offering incremental improvements for NLP researchers and practitioners.
The paper tackled the problem of pre-trained models lacking task-specific knowledge for Chinese Word Segmentation (CWS) by proposing METASEG, a model that incorporates meta learning into a multi-criteria pre-training task, achieving new state-of-the-art performance on twelve datasets and improving results in low-resource settings.
Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge and ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model METASEG, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that METASEG could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, METASEG can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.