iCub! Do you recognize what I am doing?: multimodal human action recognition on multisensory-enabled iCub robot
It addresses action recognition for robots in social tasks like partner-specific adaptation, but it is incremental as it builds on existing multimodal methods.
This study tackled human action recognition for human-robot interaction by using multisensory data from the iCub robot, showing that a multimodal ensemble learning approach improved recognition accuracy in most cases compared to single-modality models.
This study uses multisensory data (i.e., color and depth) to recognize human actions in the context of multimodal human-robot interaction. Here we employed the iCub robot to observe the predefined actions of the human partners by using four different tools on 20 objects. We show that the proposed multimodal ensemble learning leverages complementary characteristics of three color cameras and one depth sensor that improves, in most cases, recognition accuracy compared to the models trained with a single modality. The results indicate that the proposed models can be deployed on the iCub robot that requires multimodal action recognition, including social tasks such as partner-specific adaptation, and contextual behavior understanding, to mention a few.