AtomEval: Atomic Evaluation of Adversarial Claims in Fact Verification
This addresses the issue of inaccurate evaluation in fact-checking systems for researchers and practitioners, though it is incremental as it builds on existing adversarial rewriting methods.
The paper tackles the problem of evaluating adversarial claims in fact verification by introducing AtomEval, a framework that decomposes claims into atoms and scores rewrites for validity, showing it provides more reliable evaluation signals than standard metrics in experiments on the FEVER dataset.
Adversarial claim rewriting is widely used to test fact-checking systems, but standard metrics fail to capture truth-conditional consistency and often label semantically corrupted rewrites as successful. We introduce AtomEval, a validity-aware evaluation framework that decomposes claims into subject-relation-object-modifier (SROM) atoms and scores adversarial rewrites with Atomic Validity Scoring (AVS), enabling detection of factual corruption beyond surface similarity. Experiments on the FEVER dataset across representative attack strategies and LLM generators show that AtomEval provides more reliable evaluation signals in our experiments. Using AtomEval, we further analyze LLM-based adversarial generators and observe that stronger models do not necessarily produce more effective adversarial claims under validity-aware evaluation, highlighting previously overlooked limitations in current adversarial evaluation practices.