BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge
This addresses the challenge of enhancing language models with explicit commonsense knowledge for improved performance in tasks requiring such reasoning.
The paper tackles the problem of integrating commonsense knowledge into language models by introducing a method that combines BERT contextual embeddings with commonsense graph embeddings, achieving higher accuracy than BERT alone and ranking fifth on a shared task leaderboard without additional pretraining.
We introduce a simple yet effective method of integrating contextual embeddings with commonsense graph embeddings, dubbed BERT Infused Graphs: Matching Over Other embeDdings. First, we introduce a preprocessing method to improve the speed of querying knowledge bases. Then, we develop a method of creating knowledge embeddings from each knowledge base. We introduce a method of aligning tokens between two misaligned tokenization methods. Finally, we contribute a method of contextualizing BERT after combining with knowledge base embeddings. We also show BERTs tendency to correct lower accuracy question types. Our model achieves a higher accuracy than BERT, and we score fifth on the official leaderboard of the shared task and score the highest without any additional language model pretraining.