VideoA11y: Method and Dataset for Accessible Video Description
This addresses the need for better video accessibility for blind and low vision users, though it is incremental as it builds on existing multimodal models and guidelines.
The paper tackled the problem of generating accessible video descriptions for blind and low vision users by introducing VideoA11y, a method using multimodal large language models and guidelines, and curated a dataset of 40,000 videos; experiments with 387 participants showed it outperforms novice human annotations and matches trained ones in metrics like clarity and user satisfaction.
Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall short due to limitations in the quality of human annotations within training datasets, resulting in descriptions that do not fully meet BLV users' needs. To address this gap, we introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals. Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users. Rigorous experiments across 15 video categories, involving 347 sighted participants, 40 BLV participants, and seven professional describers, showed that VideoA11y descriptions outperform novice human annotations and are comparable to trained human annotations in clarity, accuracy, objectivity, descriptiveness, and user satisfaction. We evaluated models on VideoA11y-40K using both standard and custom metrics, demonstrating that MLLMs fine-tuned on this dataset produce high-quality accessible descriptions. Code and dataset are available at https://people-robots.github.io/VideoA11y.