CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations
This work addresses a specific bottleneck in Chinese NLP by enhancing semantic encoding in PLMs, offering an incremental improvement that can be easily integrated into existing models.
The authors tackled the problem of Chinese pre-trained language models lacking semantic interaction between word and character representations by proposing CLOWER, which uses contrastive learning to integrate word information into character representations, achieving superior performance on downstream tasks compared to state-of-the-art baselines.
Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines.Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.