StyleBERT: Chinese pretraining by font style information
This addresses the problem of improving Chinese NLP tasks for researchers and practitioners, but it is incremental as it adapts existing pre-training methods to Chinese-specific features.
The authors tackled the need for a Chinese pre-trained language model by proposing StyleBERT, which incorporates font style information like glyph, pinyin, five stroke, and chaizi embeddings, and it achieved strong performances on various Chinese NLP tasks.
With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get a better performance of Chinese NLP task. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.