Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
This work addresses the need for verifiable commonsense reasoning evaluation in AI, offering a dataset and baseline to improve language understanding models, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating the reasoning process behind language models' predictions by introducing the TRIP dataset, which provides dense annotations for multi-tiered evaluation, and found that while large LMs achieve high end performance, they often fail to provide valid supporting evidence.
Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning. In this paper, we highlight the importance of evaluating the underlying reasoning process in addition to end performance. Toward this goal, we introduce Tiered Reasoning for Intuitive Physics (TRIP), a novel commonsense reasoning dataset with dense annotations that enable multi-tiered evaluation of machines' reasoning process. Our empirical results show that while large LMs can achieve high end performance, they struggle to support their predictions with valid supporting evidence. The TRIP dataset and our baseline results will motivate verifiable evaluation of commonsense reasoning and facilitate future research toward developing better language understanding and reasoning models.