Impact of multiple modalities on emotion recognition: investigation into 3d facial landmarks, action units, and physiological data
This work addresses emotion recognition for applications like human-computer interaction, but it is incremental as it analyzes existing modalities without introducing new methods.
The study investigated the impact of 3D facial landmarks, action units, and physiological data on emotion recognition, finding that 3D facial landmarks and physiological data are promising individually, while action units are difficult to use alone but beneficial when fused with other modalities.
To fully understand the complexities of human emotion, the integration of multiple physical features from different modalities can be advantageous. Considering this, we present an analysis of 3D facial data, action units, and physiological data as it relates to their impact on emotion recognition. We analyze each modality independently, as well as the fusion of each for recognizing human emotion. This analysis includes which features are most important for specific emotions (e.g. happy). Our analysis indicates that both 3D facial landmarks and physiological data are encouraging for expression/emotion recognition. On the other hand, while action units can positively impact emotion recognition when fused with other modalities, the results suggest it is difficult to detect emotion using them in a unimodal fashion.