How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image
It addresses misinformation in social media by identifying deceptive thumbnails, though it is incremental as it applies an existing method to a new domain.
This study tackled the problem of fake news using misleading thumbnails by measuring semantic incongruity between images and text using CLIP, finding that fake news employs more incongruous images than general news and successfully detecting such articles.
This study investigates how fake news uses a thumbnail for a news article with a focus on whether a news article's thumbnail represents the news content correctly. A news article shared with an irrelevant thumbnail can mislead readers into having a wrong impression of the issue, especially in social media environments where users are less likely to click the link and consume the entire content. We propose to capture the degree of semantic incongruity in the multimodal relation by using the pretrained CLIP representation. From a source-level analysis, we found that fake news employs a more incongruous image to the main content than general news. Going further, we attempted to detect news articles with image-text incongruity. Evaluation experiments suggest that CLIP-based methods can successfully detect news articles in which the thumbnail is semantically irrelevant to news text. This study contributes to the research by providing a novel view on tackling online fake news and misinformation. Code and datasets are available at https://github.com/ssu-humane/fake-news-thumbnail.