Interventional Imbalanced Multi-Modal Representation Learning via $β$-Generalization Front-Door Criterion
This addresses performance degradation in multi-modal AI tasks due to modality imbalance, offering a causal solution, but it appears incremental as it builds on existing benchmark methods.
The paper tackles the problem of imbalanced contributions of modalities in multi-modal representation learning, which degrades performance, and proposes a causal approach with a novel network that achieves improved discriminative performance, supported by theoretical and empirical evaluations.
Multi-modal methods establish comprehensive superiority over uni-modal methods. However, the imbalanced contributions of different modalities to task-dependent predictions constantly degrade the discriminative performance of canonical multi-modal methods. Based on the contribution to task-dependent predictions, modalities can be identified as predominant and auxiliary modalities. Benchmark methods raise a tractable solution: augmenting the auxiliary modality with a minor contribution during training. However, our empirical explorations challenge the fundamental idea behind such behavior, and we further conclude that benchmark approaches suffer from certain defects: insufficient theoretical interpretability and limited exploration capability of discriminative knowledge. To this end, we revisit multi-modal representation learning from a causal perspective and build the Structural Causal Model. Following the empirical explorations, we determine to capture the true causality between the discriminative knowledge of predominant modality and predictive label while considering the auxiliary modality. Thus, we introduce the $β$-generalization front-door criterion. Furthermore, we propose a novel network for sufficiently exploring multi-modal discriminative knowledge. Rigorous theoretical analyses and various empirical evaluations are provided to support the effectiveness of the innate mechanism behind our proposed method.