Debiased Multimodal Understanding for Human Language Sequences
This addresses a key limitation in multimodal expression analysis for applications like sentiment detection, offering a plug-and-play solution to enhance model robustness, though it is incremental as it builds on existing causal methods.
The paper tackles the subject variation problem in multimodal language understanding, where models learn spurious correlations from subject-specific data, and proposes SuCI, a causal intervention module that disentangles subject confounders, achieving improved performance and generalizability across new subjects.
Human multimodal language understanding (MLU) is an indispensable component of expression analysis (e.g., sentiment or humor) from heterogeneous modalities, including visual postures, linguistic contents, and acoustic behaviours. Existing works invariably focus on designing sophisticated structures or fusion strategies to achieve impressive improvements. Unfortunately, they all suffer from the subject variation problem due to data distribution discrepancies among subjects. Concretely, MLU models are easily misled by distinct subjects with different expression customs and characteristics in the training data to learn subject-specific spurious correlations, limiting performance and generalizability across new subjects. Motivated by this observation, we introduce a recapitulative causal graph to formulate the MLU procedure and analyze the confounding effect of subjects. Then, we propose SuCI, a simple yet effective causal intervention module to disentangle the impact of subjects acting as unobserved confounders and achieve model training via true causal effects. As a plug-and-play component, SuCI can be widely applied to most methods that seek unbiased predictions. Comprehensive experiments on several MLU benchmarks clearly show the effectiveness of the proposed module.