ROSA: Addressing text understanding challenges in photographs via ROtated SAmpling
This addresses a practical challenge for visually impaired people by enhancing text understanding in their photos, though it is incremental as it builds on existing VQA methods.
The paper tackles the problem of Visual Question Answering (VQA) systems struggling with misaligned text in photos taken by visually impaired individuals, and introduces ROSA, a decoding strategy that improves VQA performance by 11.7 absolute points over Greedy decoding.
Visually impaired people could benefit from Visual Question Answering (VQA) systems to interpret text in their surroundings. However, current models often struggle with recognizing text in the photos taken by this population. Through in-depth interviews with visually impaired individuals, we identified common framing conventions that frequently result in misaligned text. Existing VQA benchmarks primarily feature well-oriented text captured by sighted users, under-representing these challenges. To address this gap, we introduce ROtated SAmpling (ROSA), a decoding strategy that enhances VQA performance in text-rich images with incorrectly oriented text. ROSA outperforms Greedy decoding by 11.7 absolute points in the best-performing model.