Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
This addresses the need for robust automated detection of hateful memes on the Internet, which is an incremental improvement over existing methods.
The paper tackled the problem of detecting hateful memes by proposing a robust adaptation framework for Large Multimodal Models, achieving state-of-the-art performance on six datasets and improved robustness under adversarial attacks compared to supervised fine-tuning models.
Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While Large Multimodal Models (LMMs) have shown promise in hateful meme detection, they face notable challenges like sub-optimal performance and limited out-of-domain generalization capabilities. Recent studies further reveal the limitations of both supervised fine-tuning (SFT) and in-context learning when applied to LMMs in this setting. To address these issues, we propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. Analysis reveals that our approach achieves improved robustness under adversarial attacks compared to SFT models. Experiments on six meme classification datasets show that our approach achieves state-of-the-art performance, outperforming larger agentic systems. Moreover, our method generates higher-quality rationales for explaining hateful content compared to standard SFT, enhancing model interpretability. Code available at https://github.com/JingbiaoMei/RGCL