A Survey of Knowledge Enhanced Pre-trained Models
It provides a comprehensive overview for researchers in NLP, but it is incremental as it surveys existing work without introducing new methods or results.
This survey addresses the limitations of pre-trained language models, such as poor robustness and lack of interpretability, by reviewing knowledge-enhanced pre-trained language models (KEPLMs) that improve understanding and reasoning.
Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.