Combining pre-trained language models and structured knowledge
This addresses the problem of enhancing language models with structured information for NLP researchers and practitioners, but it is incremental as it reviews existing work.
The paper surveys approaches for integrating structured knowledge (like knowledge graphs) into transformer-based language models, identifying challenges and opportunities, particularly in adapter-based injections and combining multiple methods.
In recent years, transformer-based language models have achieved state of the art performance in various NLP benchmarks. These models are able to extract mostly distributional information with some semantics from unstructured text, however it has proven challenging to integrate structured information, such as knowledge graphs into these models. We examine a variety of approaches to integrate structured knowledge into current language models and determine challenges, and possible opportunities to leverage both structured and unstructured information sources. From our survey, we find that there are still opportunities at exploiting adapter-based injections and that it may be possible to further combine various of the explored approaches into one system.