The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection
This addresses the problem of enhancing factual knowledge in language models for NLP applications, but appears incremental as it builds on existing adapter and probing methods.
The paper tackled the problem of injecting factual knowledge into large pre-trained language models by training adapter modules on parts of the ConceptNet knowledge graph using masked language modeling, and found the technique effective, increasing performance on subsets of the LAMA probe for large values of k by adding as little as 2.1% additional parameters.
This paper studies the problem of injecting factual knowledge into large pre-trained language models. We train adapter modules on parts of the ConceptNet knowledge graph using the masked language modeling objective and evaluate the success of the method by a series of probing experiments on the LAMA probe. Mean P@K curves for different configurations indicate that the technique is effective, increasing the performance on subsets of the LAMA probe for large values of k by adding as little as 2.1% additional parameters to the original models.