"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with VLMs
This addresses reliability gaps in AI tools for blind and low-vision users, though it is incremental as it focuses on evaluating existing models rather than developing new ones.
The study evaluated how image quality issues like blur and misframing affect the accuracy of Vision-Language Model (VLM) captions for product identification by blind and low-vision people, finding that accuracy dropped from 98% with no issues to 75% with issues present and worsened with compounding problems.
Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal products, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues, like blur and misframing of items, affect the accuracy of VLM-generated captions and whether resulting captions meet BLV people's information needs. Grounded in a survey with 86 BLV people, we systematically evaluate how image quality issues affect captions generated by VLMs. We show that the best model recognizes products in images with no quality issues with 98% accuracy, but drops to 75% accuracy overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.