SemanticScanpath: Combining Gaze and Speech for Situated Human-Robot Interaction Using LLMs
This work addresses the need for more intuitive and fluent interactions in social robotics by enabling robots to better interpret ambiguous verbal cues through nonverbal signals like gaze.
The paper tackled the problem of ambiguous spoken requests in human-robot interaction by integrating speech and gaze data using LLMs, resulting in a system that achieved higher situated awareness and accuracy compared to control conditions across multiple tasks and scenarios.
Large Language Models (LLMs) have substantially improved the conversational capabilities of social robots. Nevertheless, for an intuitive and fluent human-robot interaction, robots should be able to ground the conversation by relating ambiguous or underspecified spoken utterances to the current physical situation and to the intents expressed nonverbally by the user, such as through referential gaze. Here, we propose a representation that integrates speech and gaze to enable LLMs to achieve higher situated awareness and correctly resolve ambiguous requests. Our approach relies on a text-based semantic translation of the scanpath produced by the user, along with the verbal requests. It demonstrates LLMs' capabilities to reason about gaze behavior, robustly ignoring spurious glances or irrelevant objects. We validate the system across multiple tasks and two scenarios, showing its superior generality and accuracy compared to control conditions. We demonstrate an implementation on a robotic platform, closing the loop from request interpretation to execution.