Exploring the Usage of Chinese Pinyin in Pretraining
This work addresses error correction and fault tolerance for ASR-introduced errors in Chinese NLP, which is a domain-specific incremental improvement.
The paper tackles the problem of SSP errors in Chinese NLP by exploring pinyin usage in pretraining, proposing PmBERT which fuses characters and pinyin to enhance error tolerance, and shows improved robustness over SOTA models on noise-added and error-correction datasets.
Unlike alphabetic languages, Chinese spelling and pronunciation are different. Both characters and pinyin take an important role in Chinese language understanding. In Chinese NLP tasks, we almost adopt characters or words as model input, and few works study how to use pinyin. However, pinyin is essential in many scenarios, such as error correction and fault tolerance for ASR-introduced errors. Most of these errors are caused by the same or similar pronunciation words, and we refer to this type of error as SSP(the same or similar pronunciation) errors for short. In this work, we explore various ways of using pinyin in pretraining models and propose a new pretraining method called PmBERT. Our method uses characters and pinyin in parallel for pretraining. Through delicate pretraining tasks, the characters and pinyin representation are fused, which can enhance the error tolerance for SSP errors. We do comprehensive experiments and ablation tests to explore what makes a robust phonetic enhanced Chinese language model. The experimental results on both the constructed noise-added dataset and the public error-correction dataset demonstrate that our model is more robust compared to SOTA models.