When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation
This addresses a key issue in vision-language model evaluation for making visual content accessible, though it is incremental in refining existing metrics.
The paper tackles the problem of conflating length with specificity in image description evaluation, showing that people prefer more specific descriptions regardless of length and that controlling for length alone does not account for specificity differences.
Vision-language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, description specificity is often conflated with their length. We argue that these two concepts must be disentangled: descriptions can be concise yet dense with information, or lengthy yet vacuous. We define specificity relative to a contrast set, where a description is more specific to the extent that it picks out the target image better than other possible images. We construct a dataset that controls for length while varying information content, and validate that people reliably prefer more specific descriptions regardless of length. We find that controlling for length alone cannot account for differences in specificity: how the length budget is allocated makes a difference. These results support evaluation approaches that directly prioritize specificity over verbosity.