AcoustEmo: Open-Vocabulary Emotion Reasoning via Utterance-Aware Acoustic Q-Former
This work addresses the challenge of capturing subtle acoustic dynamics like micro-prosody in emotion recognition for multimodal AI systems, representing an incremental improvement over existing methods.
The paper tackles the problem of fine-grained acoustic modeling in open-vocabulary emotion recognition by proposing AcoustEmo, a time-sensitive multimodal large language model with a novel Utterance-Aware Acoustic Q-Former, which significantly enhances complex emotion reasoning on the EMER task and outperforms baselines.
Multimodal Large Language Models (MLLMs) excel in Open-Vocabulary (OV) emotion recognition but often neglect fine-grained acoustic modeling. Existing methods typically use global audio encoders, failing to capture subtle, local temporal dynamics like micro-prosody and intonation shifts within individual utterances. To address this, we propose AcoustEmo, a time-sensitive MLLM featuring a novel Utterance-Aware Acoustic Q-Former. Our approach utilizes a timestamp-synchronized sliding window to dynamically extract segment-level audio tokens instead of coarse global representations. This enables the model to explicitly trace the temporal evolution of subtle acoustic clues and capture deep contextual dependencies in dialogues. Experiments on the Explainable Multimodal Emotion Recognition (EMER) task show that AcoustEmo significantly enhances complex emotion reasoning, outperforming baselines while maintaining robust contextual accuracy.