RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning
This addresses the need for robust evaluation of mathematical reasoning in LLMs, though it is incremental as it builds on existing perturbation methods.
The paper tackled the problem of inflated performance in mathematical reasoning by large language models (LLMs) due to data leakage or pattern matching, by proposing RIDE, an adversarial question-rewriting framework that uses Item Response Theory to generate more challenging, well-posed variations, resulting in an average 21.73% performance drop across 26 advanced LLMs.
Large language models (LLMs) achieve high performance on mathematical reasoning, but these results can be inflated by training data leakage or superficial pattern matching rather than genuine reasoning. To this end, an adversarial perturbation-based evaluation is needed to measure true mathematical reasoning ability. Current rule-based perturbation methods often generate ill-posed questions and impede the systematic evaluation of question difficulty and the evolution of benchmarks. To bridge this gap, we propose RIDE, a novel adversarial question-rewriting framework that leverages Item Response Theory (IRT) to rigorously measure question difficulty and to generate intrinsically more challenging, well-posed variations of mathematical problems. We employ 35 LLMs to simulate students and build a difficulty ranker from their responses. This ranker provides a reward signal during reinforcement learning and guides a question-rewriting model to reformulate existing questions across difficulty levels. Applying RIDE to competition-level mathematical benchmarks yields perturbed versions that degrade advanced LLM performance, with experiments showing an average 21.73% drop across 26 models, thereby exposing limited robustness in mathematical reasoning and confirming the validity of our evaluation approach.