Interact with me: Joint Egocentric Forecasting of Intent to Interact, Attitude and Social Actions
This addresses the challenge of efficient human-agent interaction for robotics and AI systems, though it appears to be an incremental improvement through a novel multitask framework on augmented data.
The paper tackles the problem of enabling agents to proactively prepare for human interactions by jointly forecasting intent to interact, attitude, and social actions from egocentric video. The proposed SocialEgoNet model achieves 83.15% average accuracy across all tasks with real-time inference using only 1 second of video input.
For efficient human-agent interaction, an agent should proactively recognize their target user and prepare for upcoming interactions. We formulate this challenging problem as the novel task of jointly forecasting a person's intent to interact with the agent, their attitude towards the agent and the action they will perform, from the agent's (egocentric) perspective. So we propose \emph{SocialEgoNet} - a graph-based spatiotemporal framework that exploits task dependencies through a hierarchical multitask learning approach. SocialEgoNet uses whole-body skeletons (keypoints from face, hands and body) extracted from only 1 second of video input for high inference speed. For evaluation, we augment an existing egocentric human-agent interaction dataset with new class labels and bounding box annotations. Extensive experiments on this augmented dataset, named JPL-Social, demonstrate \emph{real-time} inference and superior performance (average accuracy across all tasks: 83.15\%) of our model outperforming several competitive baselines. The additional annotations and code will be available upon acceptance.