Know Your Source: A Public Knowledge Store for Media Background Checks
For researchers in automated fact-checking and retrieval-augmented generation, this work provides a reproducible and low-cost alternative to proprietary APIs for source-critical reasoning, addressing a key bottleneck in the field.
The paper introduces MEDIAREF, a publicly available knowledge store of web-sourced documents for generating media background checks (MBCs) to assess source credibility in automated fact-checking. It demonstrates that MEDIAREF enables reproducible, low-cost MBC generation and supports higher-quality MBCs compared to existing approaches.
LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.