Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations
This work addresses the problem of understanding and improving commonsense reasoning in NLP models for researchers and practitioners, but it is incremental as it builds on existing methods.
The paper investigates BERT's commonsense reasoning capabilities, finding it encodes many commonsense features but has deficiencies, and shows that augmenting pretraining data with minimal additional data improves downstream task performance while highlighting the importance of explicit knowledge graphs.
Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities. First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT's pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data. Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.