A Multimodal Framework for Human-Multi-Agent Interaction
This addresses the problem of enabling natural and scalable interaction in shared physical spaces for human-robot interaction, though it appears incremental as it builds on existing multimodal and LLM approaches.
The paper tackles the challenge of integrating multimodal perception, embodied expression, and coordinated decision-making in human-multi-agent interaction by introducing a framework where robots act as autonomous cognitive agents with LLM-driven planning and centralized coordination. The result is demonstrated through representative interaction runs showing coordinated multimodal reasoning and grounded embodied responses on two humanoid robots.
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.