Are Multimodal Transformers Robust to Missing Modality?
This addresses the problem of modal-incomplete data for researchers and practitioners in multimodal AI, offering an incremental improvement by automating fusion strategy selection.
The paper investigates the robustness of Transformer models to missing modalities in multimodal data, finding that they are sensitive and that optimal fusion strategies are dataset-dependent. It proposes an automated method to search for optimal fusion strategies, validated on three benchmarks.
Multimodal data collected from the real world are often imperfect due to missing modalities. Therefore multimodal models that are robust against modal-incomplete data are highly preferred. Recently, Transformer models have shown great success in processing multimodal data. However, existing work has been limited to either architecture designs or pre-training strategies; whether Transformer models are naturally robust against missing-modal data has rarely been investigated. In this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of Transformers in the presence of modal-incomplete data. Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness. What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases. Based on these findings, we propose a principle method to improve the robustness of Transformer models by automatically searching for an optimal fusion strategy regarding input data. Experimental validations on three benchmarks support the superior performance of the proposed method.