MAJIC: Leveraging Articulatory Motion for Speech-based Emotion Recognition
This work addresses the challenge of recognizing subtle emotional expressions in speech for emotion recognition systems, which typically degrade on non-acted data.
MAJIC integrates articulatory motion (jaw and facial muscle movements) with audio features using multi-task learning for speech-based emotion recognition, achieving 93% accuracy and 91% F1 score on a diverse dataset of 20 participants across 10 languages and multiple scenarios, outperforming audio-only baselines.
We introduce MAJIC, a multimodal emotion recognition system that leverages articulatory motion of the jaw and facial muscles for speech-based emotion recognition (SER). While most SER systems perform well on datasets with strongly expressed emotional speech of trained actors, their performance often degrades when emotional expressions become more subtle. We explore this challenge by engineering features from articulatory motion and integrating them with audio features using a multi-task learning framework. Our key insight is that emotion in speech manifests not only through vocal characteristics but also through distinct articulatory motions: jaw movements, facial muscle vibrations, and speech-induced vibrations. While audio captures features such as pitch and prosody, articulatory motion contains complementary information that is not present in audio alone. We evaluate our system on data collected from 20 participants across multiple sessions, 10 languages, and diverse scenarios, including prompted and conversational speech, showing its robustness across users and settings. MAJIC achieves 93% accuracy and 91% F1 score for emotion classification, outperforming strong audio-based baselines on our dataset.