INO at Factify 2: Structure Coherence based Multi-Modal Fact Verification
This work addresses the problem of combating fake news on social media for users and platforms, but it is incremental as it builds on existing methods for multi-modal fact verification.
The paper tackled multi-modal fake news detection by proposing a structure coherence-based verification scheme that combines text and image features, achieving a weighted average F1 score of 0.8079 and securing 2nd place in the FACTIFY2 challenge.
This paper describes our approach to the multi-modal fact verification (FACTIFY) challenge at AAAI2023. In recent years, with the widespread use of social media, fake news can spread rapidly and negatively impact social security. Automatic claim verification becomes more and more crucial to combat fake news. In fact verification involving multiple modal data, there should be a structural coherence between claim and document. Therefore, we proposed a structure coherence-based multi-modal fact verification scheme to classify fake news. Our structure coherence includes the following four aspects: sentence length, vocabulary similarity, semantic similarity, and image similarity. Specifically, CLIP and Sentence BERT are combined to extract text features, and ResNet50 is used to extract image features. In addition, we also extract the length of the text as well as the lexical similarity. Then the features were concatenated and passed through the random forest classifier. Finally, our weighted average F1 score has reached 0.8079, achieving 2nd place in FACTIFY2.