Seeing Through VisualBERT: A Causal Adventure on Memetic Landscapes
This work addresses the need for reliable interpretation in safety-critical applications like offensive meme detection, though it appears incremental as it builds on existing methods with causal modeling.
The paper tackled the problem of interpreting opaque deep neural networks for offensive meme detection by proposing a Structural Causal Model framework with VisualBERT, which improved transparency in understanding model behavior and misclassifications, and found that input attribution methods lack causality.
Detecting offensive memes is crucial, yet standard deep neural network systems often remain opaque. Various input attribution-based methods attempt to interpret their behavior, but they face challenges with implicitly offensive memes and non-causal attributions. To address these issues, we propose a framework based on a Structural Causal Model (SCM). In this framework, VisualBERT is trained to predict the class of an input meme based on both meme input and causal concepts, allowing for transparent interpretation. Our qualitative evaluation demonstrates the framework's effectiveness in understanding model behavior, particularly in determining whether the model was right due to the right reason, and in identifying reasons behind misclassification. Additionally, quantitative analysis assesses the significance of proposed modelling choices, such as de-confounding, adversarial learning, and dynamic routing, and compares them with input attribution methods. Surprisingly, we find that input attribution methods do not guarantee causality within our framework, raising questions about their reliability in safety-critical applications. The project page is at: https://newcodevelop.github.io/causality_adventure/