VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation
This addresses the need for better evaluation metrics in text-to-image generation, offering a multi-dimensional benchmark with high human alignment, though it is incremental in improving existing evaluation frameworks.
The paper tackles the problem of evaluating text-to-image models by introducing VISTAR, a user-centric benchmark that uses deterministic metrics and a novel HWPQ scheme, achieving over 75% human alignment and 85.9% accuracy on abstract semantics. It evaluates state-of-the-art models, showing no universal champion and providing role-weighted guidance for deployment.
We present VISTAR, a user-centric, multi-dimensional benchmark for text-to-image (T2I) evaluation that addresses the limitations of existing metrics. VISTAR introduces a two-tier hybrid paradigm: it employs deterministic, scriptable metrics for physically quantifiable attributes (e.g., text rendering, lighting) and a novel Hierarchical Weighted P/N Questioning (HWPQ) scheme that uses constrained vision-language models to assess abstract semantics (e.g., style fusion, cultural fidelity). Grounded in a Delphi study with 120 experts, we defined seven user roles and nine evaluation angles to construct the benchmark, which comprises 2,845 prompts validated by over 15,000 human pairwise comparisons. Our metrics achieve high human alignment (>75%), with the HWPQ scheme reaching 85.9% accuracy on abstract semantics, significantly outperforming VQA baselines. Comprehensive evaluation of state-of-the-art models reveals no universal champion, as role-weighted scores reorder rankings and provide actionable guidance for domain-specific deployment. All resources are publicly released to foster reproducible T2I assessment.