Relational World Knowledge Representation in Contextual Language Models: A Review
It provides a structured overview for researchers in natural language processing to understand and improve knowledge representation in LMs, but it is incremental as it reviews existing work.
This review addresses the limitations of relational knowledge bases (KBs) by examining how deep contextual language models (LMs) can internalize and express relational knowledge more flexibly, proposing a taxonomy for knowledge representation strategies based on KB supervision levels.
Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.