TurQUaz at CheckThat! 2025: Debating Large Language Models for Scientific Web Discourse Detection
This work addresses the challenge of identifying scientific content in social media for misinformation detection, though it is incremental with mixed performance.
The paper tackled the problem of detecting scientific web discourse in tweets by proposing a novel council debate method using multiple large language models, achieving first place in detecting references to scientific studies but ranking lower for other categories.
In this paper, we present our work developed for the scientific web discourse detection task (Task 4a) of CheckThat! 2025. We propose a novel council debate method that simulates structured academic discussions among multiple large language models (LLMs) to identify whether a given tweet contains (i) a scientific claim, (ii) a reference to a scientific study, or (iii) mentions of scientific entities. We explore three debating methods: i) single debate, where two LLMs argue for opposing positions while a third acts as a judge; ii) team debate, in which multiple models collaborate within each side of the debate; and iii) council debate, where multiple expert models deliberate together to reach a consensus, moderated by a chairperson model. We choose council debate as our primary model as it outperforms others in the development test set. Although our proposed method did not rank highly for identifying scientific claims (8th out of 10) or mentions of scientific entities (9th out of 10), it ranked first in detecting references to scientific studies.