Assessing the Impact of Conspiracy Theories Using Large Language Models
This work addresses the challenge of prioritizing responses to conspiracy theories, especially during crises, by applying LLMs, though it is incremental as it builds on existing methods for impact assessment.
The study tackled the problem of assessing the impact of conspiracy theories by developing datasets with human-annotated impacts and designing strategies to leverage large language models for human-like assessments, finding that multi-step reasoning improves accuracy but LLMs exhibit biases like favoring earlier-presented theories and performing poorly on emotionally charged ones.
Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.