Probing Physical Reasoning with Counter-Commonsense Context
Kazushi Kondo Saku Sugawara Akiko Aizawa
In this study, we create a CConS (Counter-commonsense Contextual Size
comparison) dataset to investigate how physical commonsense affects the
contextualized size comparison task; the proposed dataset consists of both
contexts that fit physical commonsense and those that do not. This dataset
tests the ability of language models to predict the size relationship between
objects under various contexts generated from our curated noun list and
templates. We measure the ability of several masked language models and
generative models. The results show that while large language models can use
prepositions such as ``in'' and ``into'' in the provided context to infer size
relationships, they fail to use verbs and thus make incorrect judgments led by
their prior physical commonsense.