FactAppeal: Identifying Epistemic Factual Appeals in News Media
This addresses the problem of analyzing credibility mechanisms in news media for researchers and practitioners in NLP and misinformation detection, though it is incremental as it builds on prior claim detection and verification work.
The paper tackles the problem of identifying how factual claims in news media are anchored by external sources or evidence, introducing the novel task of Epistemic Appeal Identification and presenting FactAppeal, a manually annotated dataset of 3,226 English-language news sentences. The best-performing model, based on Gemma 2 9B, achieves a macro-F1 score of 0.73.
How is a factual claim made credible? We propose the novel task of Epistemic Appeal Identification, which identifies whether and how factual statements have been anchored by external sources or evidence. To advance research on this task, we present FactAppeal, a manually annotated dataset of 3,226 English-language news sentences. Unlike prior resources that focus solely on claim detection and verification, FactAppeal identifies the nuanced epistemic structures and evidentiary basis underlying these claims and used to support them. FactAppeal contains span-level annotations which identify factual statements and mentions of sources on which they rely. Moreover, the annotations include fine-grained characteristics of factual appeals such as the type of source (e.g. Active Participant, Witness, Expert, Direct Evidence), whether it is mentioned by name, mentions of the source's role and epistemic credentials, attribution to the source via direct or indirect quotation, and other features. We model the task with a range of encoder models and generative decoder models in the 2B-9B parameter range. Our best performing model, based on Gemma 2 9B, achieves a macro-F1 score of 0.73.