STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models
This addresses a gap in evaluating subtle object state recognition for VLM researchers, providing a rigorous benchmark and large-scale training dataset, though it is incremental as it builds on existing VLM capabilities.
The paper tackles the problem of evaluating object state understanding in Vision-Language Models (VLMs) by introducing STATUS Bench, a benchmark with three tasks, and finds that current VLMs struggle significantly, with most open-weight models showing chance-level zero-shot performance, but fine-tuning on STATUS Train (13 million descriptions) allows Qwen2.5-VL to achieve performance comparable to Gemini 2.0 Flash.
Object state recognition aims to identify the specific condition of objects, such as their positional states (e.g., open or closed) and functional states (e.g., on or off). While recent Vision-Language Models (VLMs) are capable of performing a variety of multimodal tasks, it remains unclear how precisely they can identify object states. To alleviate this issue, we introduce the STAte and Transition UnderStanding Benchmark (STATUS Bench), the first benchmark for rigorously evaluating the ability of VLMs to understand subtle variations in object states in diverse situations. Specifically, STATUS Bench introduces a novel evaluation scheme that requires VLMs to perform three tasks simultaneously: object state identification (OSI), image retrieval (IR), and state change identification (SCI). These tasks are defined over our fully hand-crafted dataset involving image pairs, their corresponding object state descriptions and state change descriptions. Furthermore, we introduce a large-scale training dataset, namely STATUS Train, which consists of 13 million semi-automatically created descriptions. This dataset serves as the largest resource to facilitate further research in this area. In our experiments, we demonstrate that STATUS Bench enables rigorous consistency evaluation and reveal that current state-of-the-art VLMs still significantly struggle to capture subtle object state distinctions. Surprisingly, under the proposed rigorous evaluation scheme, most open-weight VLMs exhibited chance-level zero-shot performance. After fine-tuning on STATUS Train, Qwen2.5-VL achieved performance comparable to Gemini 2.0 Flash. These findings underscore the necessity of STATUS Bench and Train for advancing object state recognition in VLM research.