Towards Multimodal Sentiment Analysis Debiasing via Bias Purification
This addresses performance bottlenecks in sentiment analysis for multimodal AI applications, but it is incremental as it builds on existing causal methods for debiasing.
The paper tackles the problem of dataset biases in Multimodal Sentiment Analysis, such as label and context biases, by proposing a causal-based framework called MCIS that purifies biases through counterfactual inference, achieving improved performance on standard benchmarks.
Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. These harmful biases potentially mislead models to focus on statistical shortcuts and spurious correlations, causing severe performance bottlenecks. To alleviate these issues, we present a Multimodal Counterfactual Inference Sentiment (MCIS) analysis framework based on causality rather than conventional likelihood. Concretely, we first formulate a causal graph to discover harmful biases from already-trained vanilla models. In the inference phase, given a factual multimodal input, MCIS imagines two counterfactual scenarios to purify and mitigate these biases. Then, MCIS can make unbiased decisions from biased observations by comparing factual and counterfactual outcomes. We conduct extensive experiments on several standard MSA benchmarks. Qualitative and quantitative results show the effectiveness of the proposed framework.