SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
This addresses the need for real-time social support in conversations, offering a novel assistive tool for individuals in social settings, though it builds incrementally on existing LLM and AR technologies.
The paper tackles the problem of providing proactive in-situ social assistance during live interactions by introducing SocialMind, an LLM-based AR system that uses multi-modal perception to generate social suggestions, achieving 38.3% higher engagement than baselines and 95% user willingness to adopt it.
Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.