MemeFier: Dual-stage Modality Fusion for Image Meme Classification
This addresses the societal issue of hate speech proliferation online through multimodal content, but it is incremental as it builds on existing methods for meme classification.
The authors tackled the problem of detecting hate speech in image memes by proposing MemeFier, a dual-stage modality fusion architecture that incorporates external knowledge and caption supervision, achieving competitive or superior performance on benchmarks like Facebook Hateful Memes, Memotion7k, and MultiOFF.
Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on https://github.com/ckoutlis/memefier.