NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction
This work addresses critical limitations in numerical fact-checking for users relying on language models, but it is incremental as it focuses on evaluating existing models rather than proposing new solutions.
The paper tackled the problem of numerical reasoning in language models for veracity prediction by evaluating state-of-the-art models with controlled perturbations, finding accuracy drops of up to 62% and that no model was robust across all conditions.
Large language models show strong performance on knowledge intensive tasks such as fact-checking and question answering, yet they often struggle with numerical reasoning. We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs using controlled perturbations, including label-flipping probes, to test robustness. Our results indicate that even leading proprietary systems experience accuracy drops of up to 62\% under certain perturbations. No model proves to be robust across all conditions. We further find that increasing context length generally reduces accuracy, but when extended context is enriched with perturbed demonstrations, most models substantially recover. These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current language models.