Multimodal Emotion Coupling via Speech-to-Facial and Bodily Gestures in Dyadic Interaction
This work addresses the problem of improving real-time emotion detection accuracy by analyzing multimodal synchrony for researchers in affective computing and AI systems, but it is incremental as it builds on existing speech-face alignment with new data on hand gestures.
This study tackled the problem of understanding how emotional speech coordinates with facial and hand movements in dyadic interactions, finding that nonoverlapping speech increased expressivity in the lower face and mouth, with sadness showing more gestures during nonoverlap and anger suppressing them during overlaps, and predictive mapping achieved highest accuracy for prosody and MFCCs in articulatory regions.
Human emotional expression emerges through coordinated vocal, facial, and gestural signals. While speech face alignment is well established, the broader dynamics linking emotionally expressive speech to regional facial and hand motion remains critical for gaining a deeper insight into how emotional and behavior cues are communicated in real interactions. Further modulating the coordination is the structure of conversational exchange like sequential turn taking, which creates stable temporal windows for multimodal synchrony, and simultaneous speech, often indicative of high arousal moments, disrupts this alignment and impacts emotional clarity. Understanding these dynamics enhances realtime emotion detection by improving the accuracy of timing and synchrony across modalities in both human interactions and AI systems. This study examines multimodal emotion coupling using region specific motion capture from dyadic interactions in the IEMOCAP corpus. Speech features included low level prosody, MFCCs, and model derived arousal, valence, and categorical emotions (Happy, Sad, Angry, Neutral), aligned with 3D facial and hand marker displacements. Expressive activeness was quantified through framewise displacement magnitudes, and speech to gesture prediction mapped speech features to facial and hand movements. Nonoverlapping speech consistently elicited greater activeness particularly in the lower face and mouth. Sadness showed increased expressivity during nonoverlap, while anger suppressed gestures during overlaps. Predictive mapping revealed highest accuracy for prosody and MFCCs in articulatory regions while arousal and valence had lower and more context sensitive correlations. Notably, hand speech synchrony was enhanced under low arousal and overlapping speech, but not for valence.