Knowledge Prompts: Injecting World Knowledge into Language Models through Soft Prompts
This addresses the need for better knowledge integration in language models, though it is incremental as it builds on existing soft prompt methods.
The paper tackled the problem of injecting world knowledge into language models by repurposing soft prompts, resulting in soft knowledge prompts that improved performance on knowledge-intensive tasks.
Soft prompts have been recently proposed as a tool for adapting large frozen language models (LMs) to new tasks. In this work, we repurpose soft prompts to the task of injecting world knowledge into LMs. We introduce a method to train soft prompts via self-supervised learning on data from knowledge bases. The resulting soft knowledge prompts (KPs) are task independent and work as an external memory of the LMs. We perform qualitative and quantitative experiments and demonstrate that: (1) KPs can effectively model the structure of the training data; (2) KPs can be used to improve the performance of LMs in different knowledge intensive tasks.