Probing Physical Reasoning with Counter-Commonsense Context
This addresses the issue of physical reasoning in AI for researchers, but it is incremental as it focuses on a specific dataset and task.
The study tackled the problem of how physical commonsense affects language models' ability to predict size relationships between objects, finding that while large models can use prepositions like 'in' to infer sizes, they fail with verbs and make incorrect judgments due to prior commonsense.
In this study, we create a CConS (Counter-commonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; the proposed dataset consists of both contexts that fit physical commonsense and those that do not. This dataset tests the ability of language models to predict the size relationship between objects under various contexts generated from our curated noun list and templates. We measure the ability of several masked language models and generative models. The results show that while large language models can use prepositions such as ``in'' and ``into'' in the provided context to infer size relationships, they fail to use verbs and thus make incorrect judgments led by their prior physical commonsense.