Beyond the Pixels: VLM-based Evaluation of Identity Preservation in Reference-Guided Synthesis
This addresses the problem of unreliable identity evaluation in generative AI for researchers and practitioners, offering a more diagnostic and consistent method, though it is incremental as it builds on existing VLM-based approaches.
The paper tackles the challenge of evaluating identity preservation in generative models by introducing Beyond the Pixels, a hierarchical framework that uses structured VLM reasoning to assess fine-grained identity changes, demonstrating strong alignment with human judgments across four state-of-the-art models and introducing a new benchmark with 1,078 image-prompt pairs.
Evaluating identity preservation in generative models remains a critical yet unresolved challenge. Existing metrics rely on global embeddings or coarse VLM prompting, failing to capture fine-grained identity changes and providing limited diagnostic insight. We introduce Beyond the Pixels, a hierarchical evaluation framework that decomposes identity assessment into feature-level transformations. Our approach guides VLMs through structured reasoning by (1) hierarchically decomposing subjects into (type, style) -> attribute -> feature decision tree, and (2) prompting for concrete transformations rather than abstract similarity scores. This decomposition grounds VLM analysis in verifiable visual evidence, reducing hallucinations and improving consistency. We validate our framework across four state-of-the-art generative models, demonstrating strong alignment with human judgments in measuring identity consistency. Additionally, we introduce a new benchmark specifically designed to stress-test generative models. It comprises 1,078 image-prompt pairs spanning diverse subject types, including underrepresented categories such as anthropomorphic and animated characters, and captures an average of six to seven transformation axes per prompt.