Captioning Images Taken by People Who Are Blind
It addresses the problem of developing captioning algorithms for blind users, but is incremental as it primarily introduces a new dataset.
The paper tackles the lack of image captioning datasets representing real user needs by introducing VizWiz-Captions, a dataset of over 39,000 images from blind individuals with five captions each, and analyzes its characteristics and challenges for algorithms.
While an important problem in the vision community is to design algorithms that can automatically caption images, few publicly-available datasets for algorithm development directly address the interests of real users. Observing that people who are blind have relied on (human-based) image captioning services to learn about images they take for nearly a decade, we introduce the first image captioning dataset to represent this real use case. This new dataset, which we call VizWiz-Captions, consists of over 39,000 images originating from people who are blind that are each paired with five captions. We analyze this dataset to (1) characterize the typical captions, (2) characterize the diversity of content found in the images, and (3) compare its content to that found in eight popular vision datasets. We also analyze modern image captioning algorithms to identify what makes this new dataset challenging for the vision community. We publicly-share the dataset with captioning challenge instructions at https://vizwiz.org