Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models
This addresses the need for better cross-cultural evaluation of AI models in art critique, though it appears incremental as it builds on existing evaluation methods.
The researchers tackled the problem of evaluating vision-language models' ability to interpret cultural meaning in art by developing a tri-tier evaluation framework, which reduced mean absolute error by 5.2% on a held-out set of 152 samples compared to human ratings.
Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated. We present a tri-tier evaluation framework for cross-cultural art-critique assessment: Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions; and Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set. The framework outputs a calibrated cultural-understanding score for model selection and cultural-gap diagnosis, together with dimension-level diagnostics and risk indicators. We evaluate 15 VLMs on 294 expert anchors spanning six cultural traditions. Key findings are that (i) automated metrics are unreliable proxies for cultural depth, (ii) Western samples score higher than non-Western samples under our sampling and rubric, and (iii) cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration. Dataset and code are available in the supplementary materials.