Proactive Human-Robot Interaction using Visuo-Lingual Transformers
This addresses the challenge of intuitive human-robot collaboration by reducing reliance on hand-crafted triggers, though it is incremental as it builds on existing transformer methods.
The paper tackled the problem of enabling robots to proactively predict human intentions during collaboration by proposing ViLing-MMT, a vision-language multimodal transformer that uses visual and lingual cues to suggest intermediate tasks, achieving accurate scene descriptions and proactive suggestions in simulation and real-world evaluations.
Humans possess the innate ability to extract latent visuo-lingual cues to infer context through human interaction. During collaboration, this enables proactive prediction of the underlying intention of a series of tasks. In contrast, robotic agents collaborating with humans naively follow elementary instructions to complete tasks or use specific hand-crafted triggers to initiate proactive collaboration when working towards the completion of a goal. Endowing such robots with the ability to reason about the end goal and proactively suggest intermediate tasks will engender a much more intuitive method for human-robot collaboration. To this end, we propose a learning-based method that uses visual cues from the scene, lingual commands from a user and knowledge of prior object-object interaction to identify and proactively predict the underlying goal the user intends to achieve. Specifically, we propose ViLing-MMT, a vision-language multimodal transformer-based architecture that captures inter and intra-modal dependencies to provide accurate scene descriptions and proactively suggest tasks where applicable. We evaluate our proposed model in simulation and real-world scenarios.