Understanding AI Evaluation Patterns: How Different GPT Models Assess Vision-Language Descriptions
It addresses the problem of cascading biases in AI evaluation for AI developers and researchers, though it is incremental as it builds on existing models and methods.
This study analyzed how three GPT variants (GPT-4o, GPT-4o-mini, GPT-5) evaluate vision-language descriptions, revealing distinct 'evaluation personalities' such as GPT-4o-mini's consistency, GPT-4o's error detection, and GPT-5's conservatism, with all GPT models showing a consistent 2:1 bias favoring negative assessments.
As AI systems increasingly evaluate other AI outputs, understanding their assessment behavior becomes crucial for preventing cascading biases. This study analyzes vision-language descriptions generated by NVIDIA's Describe Anything Model and evaluated by three GPT variants (GPT-4o, GPT-4o-mini, GPT-5) to uncover distinct "evaluation personalities" the underlying assessment strategies and biases each model demonstrates. GPT-4o-mini exhibits systematic consistency with minimal variance, GPT-4o excels at error detection, while GPT-5 shows extreme conservatism with high variability. Controlled experiments using Gemini 2.5 Pro as an independent question generator validate that these personalities are inherent model properties rather than artifacts. Cross-family analysis through semantic similarity of generated questions reveals significant divergence: GPT models cluster together with high similarity while Gemini exhibits markedly different evaluation strategies. All GPT models demonstrate a consistent 2:1 bias favoring negative assessment over positive confirmation, though this pattern appears family-specific rather than universal across AI architectures. These findings suggest that evaluation competence does not scale with general capability and that robust AI assessment requires diverse architectural perspectives.