An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction
This work addresses the problem of enabling more natural and intuitive collaboration between humans and robots, though it is incremental as it integrates existing models into a new framework.
The paper tackles the challenge of interpreting human intent in human-robot interaction by developing a multimodal framework that combines vision-language models, speech processing, and fuzzy logic for robotic arm control, achieving a command execution accuracy of 75% in experiments.
Interpreting human intent accurately is a central challenge in human-robot interaction (HRI) and a key requirement for achieving more natural and intuitive collaboration between humans and machines. This work presents a novel multimodal HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic to enable precise and adaptive control of a Dobot Magician robotic arm. The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition, providing users with a seamless and intuitive interface for object manipulation through spoken commands. By jointly addressing scene perception and action planning, the approach enhances the reliability of command interpretation and execution. Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75\%, highlighting both the robustness and adaptability of the system. Beyond its current performance, the proposed architecture serves as a flexible and extensible foundation for future HRI research, offering a practical pathway toward more sophisticated and natural human-robot collaboration through tightly coupled speech and vision-language processing.