Improving Multimodal Sentiment Analysis via Modality Optimization and Dynamic Primary Modality Selection
This work addresses the challenge of adapting to dynamic modality importance in multimodal sentiment analysis, representing an incremental improvement over existing fixed primary modality strategies.
The paper tackled the problem of imbalanced unimodal performance and suboptimal fused representations in multimodal sentiment analysis by proposing a modality optimization and dynamic primary modality selection framework, which outperformed state-of-the-art methods on four benchmark datasets.
Multimodal Sentiment Analysis (MSA) aims to predict sentiment from language, acoustic, and visual data in videos. However, imbalanced unimodal performance often leads to suboptimal fused representations. Existing approaches typically adopt fixed primary modality strategies to maximize dominant modality advantages, yet fail to adapt to dynamic variations in modality importance across different samples. Moreover, non-language modalities suffer from sequential redundancy and noise, degrading model performance when they serve as primary inputs. To address these issues, this paper proposes a modality optimization and dynamic primary modality selection framework (MODS). First, a Graph-based Dynamic Sequence Compressor (GDC) is constructed, which employs capsule networks and graph convolution to reduce sequential redundancy in acoustic/visual modalities. Then, we develop a sample-adaptive Primary Modality Selector (MSelector) for dynamic dominance determination. Finally, a Primary-modality-Centric Cross-Attention (PCCA) module is designed to enhance dominant modalities while facilitating cross-modal interaction. Extensive experiments on four benchmark datasets demonstrate that MODS outperforms state-of-the-art methods, achieving superior performance by effectively balancing modality contributions and eliminating redundant noise.