EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis
This work addresses emotion analysis from multimodal signals, which is an incremental improvement in a domain-specific area.
The paper tackles the challenge of decoding emotions from complex multimodal interactions by designing three types of multimodal latent representations and a hierarchical fusion method, resulting in a model that outperforms state-of-the-art methods with lower computing complexity and fewer trainable parameters.
Humans are skilled in reading the interlocutor's emotion from multimodal signals, including spoken words, simultaneous speech, and facial expressions. It is still a challenge to effectively decode emotions from the complex interactions of multimodal signals. In this paper, we design three kinds of multimodal latent representations to refine the emotion analysis process and capture complex multimodal interactions from different views, including a intact three-modal integrating representation, a modality-shared representation, and three modality-individual representations. Then, a modality-semantic hierarchical fusion is proposed to reasonably incorporate these representations into a comprehensive interaction representation. The experimental results demonstrate that our EffMulti outperforms the state-of-the-art methods. The compelling performance benefits from its well-designed framework with ease of implementation, lower computing complexity, and less trainable parameters.