Evaluating Commonsense in Pre-trained Language Models
This work addresses the gap in understanding commonsense in language models, which is crucial for human-like question answering and reading comprehension, though it is incremental as it builds on existing evaluation frameworks.
The study evaluated the commonsense knowledge of pre-trained language models like GPT, BERT, XLNet, and RoBERTa on seven benchmarks, finding that language modeling objectives enhance commonsense ability, but models struggle with tasks requiring multiple inference steps and show surface-level learning.
Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.