VALUED -- Vision and Logical Understanding Evaluation Dataset
This addresses the need for datasets to evaluate logical consistency in vision models, which is crucial for critical applications, though it is incremental as it builds on existing efforts to improve model robustness.
The authors tackled the problem of deep learning models failing to capture semantic context and logical constraints by introducing the VALUE dataset, which consists of over 200,000 annotated images based on chess with curated rules, and they showed that state-of-the-art vision models produce many incoherent results despite good standard metric performance.
Starting with early successes in computer vision tasks, deep learning based techniques have since overtaken state of the art approaches in a multitude of domains. However, it has been demonstrated time and again that these techniques fail to capture semantic context and logical constraints, instead often relying on spurious correlations to arrive at the answer. Since application of deep learning techniques to critical scenarios are dependent on adherence to domain specific constraints, several attempts have been made to address this issue. One limitation holding back a thorough exploration of this area, is a lack of suitable datasets which feature a rich set of rules. In order to address this, we present the VALUE (Vision And Logical Understanding Evaluation) Dataset, consisting of 200,000$+$ annotated images and an associated rule set, based on the popular board game - chess. The curated rule set considerably constrains the set of allowable predictions, and are designed to probe key semantic abilities like localization and enumeration. Alongside standard metrics, additional metrics to measure performance with regards to logical consistency is presented. We analyze several popular and state of the art vision models on this task, and show that, although their performance on standard metrics are laudable, they produce a plethora of incoherent results, indicating that this dataset presents a significant challenge for future works.