Evaluating Evidence Attribution in Generated Fact Checking Explanations
This work addresses trustworthiness issues in automated fact-checking systems for users relying on generated explanations, though it is incremental as it focuses on evaluation rather than a new generation method.
The paper tackles the problem of hallucinations in automated fact-checking explanations by introducing a citation masking and recovery evaluation protocol to assess evidence attribution quality, finding that even top LLMs produce inaccurate attributions and human-curated evidence is crucial for better explanations.
Automated fact-checking systems often struggle with trustworthiness, as their generated explanations can include hallucinations. In this work, we explore evidence attribution for fact-checking explanation generation. We introduce a novel evaluation protocol -- citation masking and recovery -- to assess attribution quality in generated explanations. We implement our protocol using both human annotators and automatic annotators, and find that LLM annotation correlates with human annotation, suggesting that attribution assessment can be automated. Finally, our experiments reveal that: (1) the best-performing LLMs still generate explanations with inaccurate attributions; and (2) human-curated evidence is essential for generating better explanations. Code and data are available here: https://github.com/ruixing76/Transparent-FCExp.