FiLMing Multimodal Sarcasm Detection with Attention
This work addresses sarcasm detection for NLP applications like opinion mining, offering a multimodal approach that is incremental but with strong performance gains.
The paper tackled sarcasm detection in multimodal social media data by proposing a novel architecture combining RoBERTa with co-attention and FiLMed ResNet blocks, achieving a 6.14% F1 score improvement over state-of-the-art methods on a Twitter dataset.
Sarcasm detection identifies natural language expressions whose intended meaning is different from what is implied by its surface meaning. It finds applications in many NLP tasks such as opinion mining, sentiment analysis, etc. Today, social media has given rise to an abundant amount of multimodal data where users express their opinions through text and images. Our paper aims to leverage multimodal data to improve the performance of the existing systems for sarcasm detection. So far, various approaches have been proposed that uses text and image modality and a fusion of both. We propose a novel architecture that uses the RoBERTa model with a co-attention layer on top to incorporate context incongruity between input text and image attributes. Further, we integrate feature-wise affine transformation by conditioning the input image through FiLMed ResNet blocks with the textual features using the GRU network to capture the multimodal information. The output from both the models and the CLS token from RoBERTa is concatenated and used for the final prediction. Our results demonstrate that our proposed model outperforms the existing state-of-the-art method by 6.14% F1 score on the public Twitter multimodal sarcasm detection dataset.