MATER: Multi-level Acoustic and Textual Emotion Representation for Interpretable Speech Emotion Recognition
This work addresses speech emotion recognition for applications in human-computer interaction, but it is incremental as it builds on existing multi-modal approaches.
The paper tackles categorical emotion recognition and emotional attribute prediction in natural speech by proposing MATER, a hierarchical framework integrating acoustic and textual features, achieving a Macro-F1 of 41.01% and an average CCC of 0.5928, with a CCC of 0.6941 in valence prediction.
This paper presents our contributions to the Speech Emotion Recognition in Naturalistic Conditions (SERNC) Challenge, where we address categorical emotion recognition and emotional attribute prediction. To handle the complexities of natural speech, including intra- and inter-subject variability, we propose Multi-level Acoustic-Textual Emotion Representation (MATER), a novel hierarchical framework that integrates acoustic and textual features at the word, utterance, and embedding levels. By fusing low-level lexical and acoustic cues with high-level contextualized representations, MATER effectively captures both fine-grained prosodic variations and semantic nuances. Additionally, we introduce an uncertainty-aware ensemble strategy to mitigate annotator inconsistencies, improving robustness in ambiguous emotional expressions. MATER ranks fourth in both tasks with a Macro-F1 of 41.01% and an average CCC of 0.5928, securing second place in valence prediction with an impressive CCC of 0.6941.