Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being?
This work addresses the need for better evaluation metrics in vision-language research, though it is incremental as it builds on existing methods to assess model biases.
The authors tackled the problem of evaluating whether vision-language models understand images like humans by proposing a quantitative metric called 'Equivariance Score' and a dataset named 'Human Puzzle', finding that models show biases toward local contexts rather than interpreting global context.
As computer vision and NLP make progress, Vision-Language(VL) is becoming an important area of research. Despite the importance, evaluation metrics of the research domain is still at a preliminary stage of development. In this paper, we propose a quantitative metric "Equivariance Score" and evaluation dataset "Human Puzzle" to assess whether a VL model is understanding an image like a human. We observed that the VL model does not interpret the overall context of an input image but instead shows biases toward a specific object or shape that forms the local context. We aim to quantitatively measure a model's performance in understanding context. To verify the current existing VL model's capability, we sliced the original input image into pieces and randomly placed them, distorting the global context of the image. Our paper discusses each VL model's level of interpretation on global context and addresses how the structural characteristics influenced the results.