Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks
This work addresses modality bias in misinformation detection for researchers, but it is incremental as it builds on existing bias quantification methods by focusing on sample-level analysis.
The paper tackled the problem of modality bias in multimodal misinformation benchmarks by proposing automated sample-specific analysis methods, revealing that ensembling multiple views is crucial for reliable analysis and that automated methods are prone to detector-induced fluctuations.
Numerous multimodal misinformation benchmarks exhibit bias toward specific modalities, allowing detectors to make predictions based solely on one modality. While previous research has quantified bias at the dataset level or manually identified spurious correlations between modalities and labels, these approaches lack meaningful insights at the sample level and struggle to scale to the vast amount of online information. In this paper, we investigate the design for automated recognition of modality bias at the sample level. Specifically, we propose three bias quantification methods based on theories/views of different levels of granularity: 1) a coarse-grained evaluation of modality benefit; 2) a medium-grained quantification of information flow; and 3) a fine-grained causality analysis. To verify the effectiveness, we conduct a human evaluation on two popular benchmarks. Experimental results reveal three interesting findings that provide potential direction toward future research: 1)~Ensembling multiple views is crucial for reliable automated analysis; 2)~Automated analysis is prone to detector-induced fluctuations; and 3)~Different views produce a higher agreement on modality-balanced samples but diverge on biased ones.