GRAFFORD: A Benchmark Dataset for Testing the Knowledge of Object
  Affordances of Language and Vision Models
Sayantan Adak Daivik Agrawal Animesh Mukherjee Somak Aditya
We investigate the knowledge of object affordances in pre-trained language
models (LMs) and pre-trained Vision-Language models (VLMs). Transformers-based
large pre-trained language models (PTLM) learn contextual representation from
massive amounts of unlabeled text and are shown to perform impressively in
downstream NLU tasks. In parallel, a growing body of literature shows that
PTLMs fail inconsistently and non-intuitively, showing a lack of reasoning and
grounding. To take a first step toward quantifying the effect of grounding (or
lack thereof), we curate a novel and comprehensive dataset of object
affordances -- GrAFFORD, characterized by 15 affordance classes. Unlike
affordance datasets collected in vision and language domains, we annotate
in-the-wild sentences with objects and affordances. Experimental results reveal
that PTLMs exhibit limited reasoning abilities when it comes to uncommon object
affordances. We also observe that pre-trained VLMs do not necessarily capture
object affordances effectively. Through few-shot fine-tuning, we demonstrate
improvement in affordance knowledge in PTLMs and VLMs. Our research contributes
a novel dataset for language grounding tasks, and presents insights into LM
capabilities, advancing the understanding of object affordances. Codes and data
are available at https://github.com/sayantan11995/Affordance