FPM: A Collection of Large-scale Foundation Pre-trained Language Models
This work provides baseline models for future research in large-scale language model pretraining, though it is incremental as it builds on existing architectures and technologies.
The paper tackles the lack of unified large-scale Transformer models by creating a collection of foundation pre-trained language models, showing that scaling up existing architectures with proper training techniques consistently improves performance, and providing the largest Chinese generative and encoding models along with English BERT models that achieve a 14.45% higher F1 score than Turing-NLR.
Large-scale Transformer models have significantly promoted the recent development of natural language processing applications. However, little effort has been made to unify the effective models. In this paper, driven by providing a new set of baseline models in the future, we adopt various novel transformer architectures and launch a model set with the help of recent mainstream technologies. We focus the discussions on optimizing the depth of the networks based on the existing powerful encode-decoder structures. We show that by properly avoiding training defects such as non-convergence and degradation, scaling up off-the-shelf transformer architectures consistently delivers better performance. To stimulate future research on large-scale language model pretraining, we present extensive results and detailed discussions on network performance improvements with respect to the network depth and confirm the existence of the optimal number of layers under specific tasks. To the best of our knowledge, we provide the largest Chinese generative model and the largest Chinese encoding model. The BERT language models we trained on English datasets deliver a 14.45% higher F1 score than the Turing-NLR.