Why Do Masked Neural Language Models Still Need Common Sense Knowledge?
This addresses a key limitation in NLP models for tasks like question answering, though it is incremental by building on existing MNLM methods.
The paper tackles the problem of masked neural language models (MNLMs) lacking common sense knowledge, showing through tests that they partially understand but often fail in semantic relations and are vulnerable in tasks requiring common sense, and demonstrates that combining external knowledge repositories can improve performance.
Currently, contextualized word representations are learned by intricate neural network models, such as masked neural language models (MNLMs). The new representations significantly enhanced the performance in automated question answering by reading paragraphs. However, identifying the detailed knowledge trained in the MNLMs is difficult owing to numerous and intermingled parameters. This paper provides empirical but insightful analyses on the pretrained MNLMs with respect to common sense knowledge. First, we propose a test that measures what types of common sense knowledge do pretrained MNLMs understand. From the test, we observed that MNLMs partially understand various types of common sense knowledge but do not accurately understand the semantic meaning of relations. In addition, based on the difficulty of the question-answering task problems, we observed that pretrained MLM-based models are still vulnerable to problems that require common sense knowledge. We also experimentally demonstrated that we can elevate existing MNLM-based models by combining knowledge from an external common sense repository.