reStructured Pre-training
This work addresses the challenge of enhancing NLP model efficiency and performance for tasks like standardized testing, offering a potentially new paradigm rather than an incremental improvement.
The paper tackles the problem of improving NLP task performance by proposing a new learning paradigm called reStructured Pre-training (RST), which emphasizes data restructuring for better storage and access during pre-training and fine-tuning. The result is that RST models outperform competitors on 52 out of 55 datasets and achieve high scores in the Gaokao-English exam, with Qin scoring 138.5 out of 150 in 2018 and 134 in 2022, surpassing GPT-3 by 15 and 26 points respectively while using fewer parameters.
In this work, we try to decipher the internal connection of NLP technology development in the past decades, searching for essence, which rewards us with a (potential) new learning paradigm for NLP tasks, dubbed as reStructured Pre-training (RST). In such a paradigm, the role of data will be re-emphasized, and model pre-training and fine-tuning of downstream tasks are viewed as a process of data storing and accessing. Based on that, we operationalize the simple principle that a good storage mechanism should not only have the ability to cache a large amount of data but also consider the ease of access. We achieve this by pre-training models over restructured data that consist of a variety of valuable information instead of raw data after overcoming several engineering challenges. Experimentally, RST models not only surpass strong competitors (e.g., T0) on 52/55 popular datasets from a variety of NLP tasks, but also achieve superior performance in National College Entrance Examination - English (Gaokao-English),the most authoritative examination in China. Specifically, the proposed system Qin achieves 40 points higher than the average scores made by students and 15 points higher than GPT3 with 1/16 parameters. In particular, Qin gets a high score of 138.5 (the full mark is 150) in the 2018 English exam (national paper III). We have released the Gaokao Benchmark with an online submission platform. In addition, we test our model in the 2022 College Entrance Examination English that happened a few days ago (2022.06.08), and it gets a total score of 134 (v.s. GPT3's 108).