Pre-trained Models for Natural Language Processing: A Survey
It serves as a hands-on guide for researchers and practitioners to understand and apply PTMs in NLP, but it is incremental as it synthesizes existing knowledge rather than introducing new methods.
This survey provides a comprehensive review of pre-trained models (PTMs) in natural language processing, categorizing existing models and outlining adaptation methods for downstream tasks.
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.