Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition
This work addresses parameter conflicts in multi-task learning for multimodal affective computing, offering an incremental improvement over existing methods.
The paper tackled the problem of parameter conflicts in multi-task learning for multimodal sentiment analysis and emotion recognition by introducing a Multimodal Mixture of Low-Rank Experts method, which achieved state-of-the-art performance on sentiment analysis and competitive results on emotion recognition benchmarks.
Multi-task learning (MTL) enables the efficient transfer of extra knowledge acquired from other tasks. The high correlation between multimodal sentiment analysis (MSA) and multimodal emotion recognition (MER) supports their joint training. However, existing methods primarily employ hard parameter sharing, ignoring parameter conflicts caused by complex task correlations. In this paper, we present a novel MTL method for MSA and MER, termed Multimodal Mixture of Low-Rank Experts (MMoLRE). MMoLRE utilizes shared and task-specific experts to distinctly model common and unique task characteristics, thereby avoiding parameter conflicts. Additionally, inspired by low-rank structures in the Mixture of Experts (MoE) framework, we design low-rank expert networks to reduce parameter and computational overhead as the number of experts increases. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MMoLRE achieves state-of-the-art performance on the MSA task and competitive results on the MER task.