AIVA: An AI-based Virtual Companion for Emotion-aware Interaction
This work addresses the problem of enhancing immersive and empathetic human-computer interaction for applications in companion robotics, social care, and mental health, representing an incremental advancement by combining existing technologies with new multimodal components.
The paper tackles the limitation of LLMs in interpreting emotional cues from non-verbal signals by proposing AIVA, an AI-based virtual companion that integrates multimodal sentiment perception for emotion-aware interactions, achieving improved empathetic responses through a novel framework.
Recent advances in Large Language Models (LLMs) have significantly improved natural language understanding and generation, enhancing Human-Computer Interaction (HCI). However, LLMs are limited to unimodal text processing and lack the ability to interpret emotional cues from non-verbal signals, hindering more immersive and empathetic interactions. This work explores integrating multimodal sentiment perception into LLMs to create emotion-aware agents. We propose \ours, an AI-based virtual companion that captures multimodal sentiment cues, enabling emotionally aligned and animated HCI. \ours introduces a Multimodal Sentiment Perception Network (MSPN) using a cross-modal fusion transformer and supervised contrastive learning to provide emotional cues. Additionally, we develop an emotion-aware prompt engineering strategy for generating empathetic responses and integrate a Text-to-Speech (TTS) system and animated avatar module for expressive interactions. \ours provides a framework for emotion-aware agents with applications in companion robotics, social care, mental health, and human-centered AI.