Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies
This work addresses misinformation detection for fact-checking applications, but it is incremental as it builds on existing VLM methods with a new classifier and fusion approach.
The study tackled multimodal fact-checking by evaluating Vision Language Models (VLMs) and proposing a probing classifier with embedding strategies, showing that fusing separate text and image embeddings outperformed VLM embeddings and the neural classifier beat baselines like KNN and SVM on two datasets.
This study evaluates the effectiveness of Vision Language Models (VLMs) in representing and utilizing multimodal content for fact-checking. To be more specific, we investigate whether incorporating multimodal content improves performance compared to text-only models and how well VLMs utilize text and image information to enhance misinformation detection. Furthermore we propose a probing classifier based solution using VLMs. Our approach extracts embeddings from the last hidden layer of selected VLMs and inputs them into a neural probing classifier for multi-class veracity classification. Through a series of experiments on two fact-checking datasets, we demonstrate that while multimodality can enhance performance, fusing separate embeddings from text and image encoders yielded superior results compared to using VLM embeddings. Furthermore, the proposed neural classifier significantly outperformed KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking.