An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
This work addresses the problem of optimizing knowledge graph-based self-supervision for language models, which is incremental as it refines existing methods for better performance on specific task properties.
The paper investigates how to select and combine knowledge from large knowledge graphs for self-supervision to improve language models, finding that encoder-decoder models benefit from more data and balanced sampling strategies yield the best performance, with most improvement on questions with short answers and dissimilar answer candidates.
Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models, in zero-shot evaluation on various downstream language reasoning tasks. Since these improvements are reported in aggregate, however, little is known about (i) how to select the appropriate knowledge for solid performance across tasks, (ii) how to combine this knowledge with neural language models, and (iii) how these pairings affect granular task performance. In this paper, we study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models. We study the effect of different synthetic datasets on language models with various architectures and sizes. The resulting models are evaluated against four task properties: domain overlap, answer similarity, vocabulary overlap, and answer length. Our experiments show that encoder-decoder models benefit from more data to learn from, whereas sampling strategies that balance across different aspects yield best performance. Most of the improvement occurs on questions with short answers and dissimilar answer candidates, which corresponds to the characteristics of the data used for pre-training.