Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Content
This addresses misinformation mitigation in social media by improving credibility prediction and interpretation, though it appears incremental as it builds on existing LLM capabilities.
The researchers tackled the challenge of predicting perceived credibility of visual content on social media by developing an LLM-informed feature discovery framework that extracts interpretable features using targeted prompts. Their method outperformed zero-shot GPT-based predictions by 13% in R2 on a dataset of 4,191 posts across eight topics.
In today's visually dominated social media landscape, predicting the perceived credibility of visual content and understanding what drives human judgment are crucial for countering misinformation. However, these tasks are challenging due to the diversity and richness of visual features. We introduce a Large Language Model (LLM)-informed feature discovery framework that leverages multimodal LLMs, such as GPT-4o, to evaluate content credibility and explain its reasoning. We extract and quantify interpretable features using targeted prompts and integrate them into machine learning models to improve credibility predictions. We tested this approach on 4,191 visual social media posts across eight topics in science, health, and politics, using credibility ratings from 5,355 crowdsourced workers. Our method outperformed zero-shot GPT-based predictions by 13 percent in R2, and revealed key features like information concreteness and image format. We discuss the implications for misinformation mitigation, visual credibility, and the role of LLMs in social science.