UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models
This addresses the lack of a unified method for knowledge injection in NLP, offering incremental improvements for tasks requiring external knowledge.
The paper tackles the problem of injecting both structured and unstructured knowledge into pre-trained language models by proposing UNTER, a unified knowledge interface, which results in continuous improvements on knowledge-driven NLP tasks like entity typing, especially in low-resource scenarios.
Recent research demonstrates that external knowledge injection can advance pre-trained language models (PLMs) in a variety of downstream NLP tasks. However, existing knowledge injection methods are either applicable to structured knowledge or unstructured knowledge, lacking a unified usage. In this paper, we propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. In UNTER, we adopt the decoder as a unified knowledge interface, aligning span representations obtained from the encoder with their corresponding knowledge. This approach enables the encoder to uniformly invoke span-related knowledge from its parameters for downstream applications. Experimental results show that, with both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks, including entity typing, named entity recognition and relation extraction, especially in low-resource scenarios.