ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
This work addresses the need for more comprehensive pre-training in natural language processing, offering incremental improvements over existing models like BERT and XLNet.
The paper tackled the problem of extracting lexical, syntactic, and semantic information from training corpora for language understanding by proposing ERNIE 2.0, a continual pre-training framework, which outperformed BERT and XLNet on 16 tasks including GLUE benchmarks and Chinese tasks.
Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.