VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models
This addresses the challenge of assisting visually impaired individuals with daily tasks, but it is incremental as it benchmarks existing methods rather than proposing new ones.
The paper tackles the problem of using large models (LMs) for visually impaired assistance (VIA) by conducting a survey and benchmark, finding that while LMs show potential, they often fail to ground responses in the environment (25.7% for GPT-4) and lack fine-grained guidance (32.1% for GPT-4).
Visually Impaired Assistance (VIA) aims to automatically help the visually impaired (VI) handle daily activities. The advancement of VIA primarily depends on developments in Computer Vision (CV) and Natural Language Processing (NLP), both of which exhibit cutting-edge paradigms with large models (LMs). Furthermore, LMs have shown exceptional multimodal abilities to tackle challenging physically-grounded tasks such as embodied robots. To investigate the potential and limitations of state-of-the-art (SOTA) LMs' capabilities in VIA applications, we present an extensive study for the task of VIA with LMs (VIALM). In this task, given an image illustrating the physical environments and a linguistic request from a VI user, VIALM aims to output step-by-step guidance to assist the VI user in fulfilling the request grounded in the environment. The study consists of a survey reviewing recent LM research and benchmark experiments examining selected LMs' capabilities in VIA. The results indicate that while LMs can potentially benefit VIA, their output cannot be well environment-grounded (i.e., 25.7% GPT-4's responses) and lacks fine-grained guidance (i.e., 32.1% GPT-4's responses).