ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
This work addresses the need for more robust and adaptable text encoders in NLP, offering incremental improvements through extended training and adaptation techniques.
The paper tackles the problem of enhancing pre-trained text encoders by integrating n-gram knowledge and scaling training with large data, resulting in new state-of-the-art performance across various NLP tasks in multiple languages and domains.
Pre-trained text encoders have drawn sustaining attention in natural language processing (NLP) and shown their capability in obtaining promising results in different tasks. Recent studies illustrated that external self-supervised signals (or knowledge extracted by unsupervised learning, such as n-grams) are beneficial to provide useful semantic evidence for understanding languages such as Chinese, so as to improve the performance on various downstream tasks accordingly. To further enhance the encoders, in this paper, we propose to pre-train n-gram-enhanced encoders with a large volume of data and advanced techniques for training. Moreover, we try to extend the encoder to different languages as well as different domains, where it is confirmed that the same architecture is applicable to these varying circumstances and new state-of-the-art performance is observed from a long list of NLP tasks across languages and domains.