Multi-Modal Sentiment Analysis with Dynamic Attention Fusion
This work addresses the need for more robust sentiment prediction in affective computing applications, such as emotion recognition and mental health assessment, by effectively integrating verbal and non-verbal cues, though it is incremental as it builds on existing encoders without finetuning.
The paper tackled the problem of unimodal sentiment analysis by introducing Dynamic Attention Fusion (DAF), a lightweight framework that combines text and acoustic features with adaptive attention, resulting in consistent performance gains over baselines on a large multimodal benchmark, including notable improvements in F1-score and reductions in prediction error.
Traditional sentiment analysis has long been a unimodal task, relying solely on text. This approach overlooks non-verbal cues such as vocal tone and prosody that are essential for capturing true emotional intent. We introduce Dynamic Attention Fusion (DAF), a lightweight framework that combines frozen text embeddings from a pretrained language model with acoustic features from a speech encoder, using an adaptive attention mechanism to weight each modality per utterance. Without any finetuning of the underlying encoders, our proposed DAF model consistently outperforms both static fusion and unimodal baselines on a large multimodal benchmark. We report notable gains in F1-score and reductions in prediction error and perform a variety of ablation studies that support our hypothesis that the dynamic weighting strategy is crucial for modeling emotionally complex inputs. By effectively integrating verbal and non-verbal information, our approach offers a more robust foundation for sentiment prediction and carries broader impact for affective computing applications -- from emotion recognition and mental health assessment to more natural human computer interaction.