R2H: Building Multimodal Navigation Helpers that Respond to Help Requests
This work addresses the need for accessible navigation tools for individuals with disabilities by proposing a benchmark to advance multimodal helper agents, though it is incremental as it builds on existing datasets and methods.
The authors tackled the problem of developing multimodal navigation helpers that can respond to help requests by introducing the R2H benchmark, which includes tasks for response generation and interaction evaluation, and they explored methods like fine-tuning a model and using a large language model, achieving results assessed through automatic and human evaluations.
Intelligent navigation-helper agents are critical as they can navigate users in unknown areas through environmental awareness and conversational ability, serving as potential accessibility tools for individuals with disabilities. In this work, we first introduce a novel benchmark, Respond to Help Requests (R2H), to promote the development of multi-modal navigation helpers capable of responding to requests for help, utilizing existing dialog-based embodied datasets. R2H mainly includes two tasks: (1) Respond to Dialog History (RDH), which assesses the helper agent's ability to generate informative responses based on a given dialog history, and (2) Respond during Interaction (RdI), which evaluates the effectiveness and efficiency of the response during consistent cooperation with a task performer. Furthermore, we explore two approaches to construct the navigation-helper agent, including fine-tuning a novel task-oriented multi-modal response generation model that can see and respond, named SeeRee, and employing a multi-modal large language model in a zero-shot manner. Analysis of the task and method was conducted based on both automatic benchmarking and human evaluations. Project website: https://sites.google.com/view/response2helprequests/home.