mmWalk: Towards Multi-modal Multi-view Walking Assistance
This addresses outdoor navigation for people with blindness or low vision, but it is incremental as it focuses on dataset creation and benchmarking.
The paper tackles the challenge of walking assistance for people with blindness or low vision by creating mmWalk, a simulated multi-modal dataset with 120 trajectories and 62k frames, and shows that state-of-the-art vision-language models struggle on its tasks, with a finetuned model validated on real-world data.
Walking assistance in extreme or complex environments remains a significant challenge for people with blindness or low vision (BLV), largely due to the lack of a holistic scene understanding. Motivated by the real-world needs of the BLV community, we build mmWalk, a simulated multi-modal dataset that integrates multi-view sensor and accessibility-oriented features for outdoor safe navigation. Our dataset comprises 120 manually controlled, scenario-categorized walking trajectories with 62k synchronized frames. It contains over 559k panoramic images across RGB, depth, and semantic modalities. Furthermore, to emphasize real-world relevance, each trajectory involves outdoor corner cases and accessibility-specific landmarks for BLV users. Additionally, we generate mmWalkVQA, a VQA benchmark with over 69k visual question-answer triplets across 9 categories tailored for safe and informed walking assistance. We evaluate state-of-the-art Vision-Language Models (VLMs) using zero- and few-shot settings and found they struggle with our risk assessment and navigational tasks. We validate our mmWalk-finetuned model on real-world datasets and show the effectiveness of our dataset for advancing multi-modal walking assistance.