TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text
This work addresses the problem of grounding in language models for researchers in AI and NLP, but it is incremental as it builds on existing studies of model limitations and introduces a new dataset.
The authors investigated the ability of pre-trained language models and vision-language models to predict object affordances from text, finding that these models exhibit limited reasoning for uncommon affordances and that vision-language models do not effectively capture such knowledge, with few-shot fine-tuning showing improvements.
We investigate the knowledge of object affordances in pre-trained language models (LMs) and pre-trained Vision-Language models (VLMs). A growing body of literature shows that PTLMs fail inconsistently and non-intuitively, demonstrating a lack of reasoning and grounding. To take a first step toward quantifying the effect of grounding (or lack thereof), we curate a novel and comprehensive dataset of object affordances -- Text2Afford, characterized by 15 affordance classes. Unlike affordance datasets collected in vision and language domains, we annotate in-the-wild sentences with objects and affordances. Experimental results reveal that PTLMs exhibit limited reasoning abilities when it comes to uncommon object affordances. We also observe that pre-trained VLMs do not necessarily capture object affordances effectively. Through few-shot fine-tuning, we demonstrate improvement in affordance knowledge in PTLMs and VLMs. Our research contributes a novel dataset for language grounding tasks, and presents insights into LM capabilities, advancing the understanding of object affordances. Codes and data are available at https://github.com/sayantan11995/Text2Afford