LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
This addresses the problem of evaluating detailed image captions for researchers and developers, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the lack of standardized evaluation for detailed image captions by introducing LOTUS, a leaderboard that assesses caption quality, risks, and societal biases, revealing no single model excels across all criteria and that optimal model selection depends on user priorities.
Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.