Preconditioned Visual Language Inference with Weak Supervision
This work addresses a gap in evaluating visual language models for commonsense reasoning, which is incremental as it builds on existing NLP studies but applies it to visual contexts.
The paper introduced the task of preconditioned visual language inference and rationalization (PVLIR) to assess if state-of-the-art visual language models can extract contextual preconditions from images to infer object affordances, revealing their shortcomings and proposing a roadmap for improvement.
Humans can infer the affordance of objects by extracting related contextual preconditions for each scenario. For example, upon seeing an image of a broken cup, we can infer that this precondition prevents the cup from being used for drinking. Reasoning with preconditions of commonsense is studied in NLP where the model explicitly gets the contextual precondition. However, it is unclear if SOTA visual language models (VLMs) can extract such preconditions and infer the affordance of objects with them. In this work, we introduce the task of preconditioned visual language inference and rationalization (PVLIR). We propose a learning resource based on three strategies to retrieve weak supervision signals for the task and develop a human-verified test set for evaluation. Our results reveal the shortcomings of SOTA VLM models in the task and draw a road map to address the challenges ahead in improving them.