Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models
This addresses a specific challenge in human-robot interaction for robotics applications, but it is incremental as it applies existing models to a known bottleneck.
The paper tackled the problem of determining when a robot should initiate communication in human-robot interaction by testing large language models (LLMs) and vision language models (VLMs) on 84 situations, finding they handle clear actions well but struggle with open-ended scenarios.
In human-robot interaction (HRI), the beginning of an interaction is often complex. Whether the robot should communicate with the human is dependent on several situational factors (e.g., the current human's activity, urgency of the interaction, etc.). We test whether large language models (LLM) and vision language models (VLM) can provide solutions to this problem. We compare four different system-design patterns using LLMs and VLMs, and test on a test set containing 84 human-robot situations. The test set mixes several publicly available datasets and also includes situations where the appropriate action to take is open-ended. Our results using the GPT-4o and Phi-3 Vision model indicate that LLMs and VLMs are capable of handling interaction beginnings when the desired actions are clear, however, challenge remains in the open-ended situations where the model must balance between the human and robot situation.