MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorization
This addresses the issue of overfitting in mathematical benchmarks for AI researchers, though it is incremental as it builds on existing benchmark methods.
The paper tackled the problem of evaluating mathematical reasoning without contamination from memorization by proposing MatheMagic, a dynamic benchmark that alters number and operator interpretations to generate test instances at runtime, revealing that models solve deduction tasks more easily than induction but fail to generalize reasoning skills.
Conducting contamination-free evaluation of mathematical capabilities can be difficult for two reasons: models may memorize a test set once it is made public, and current mathematical benchmarks are prone to overfitting due to having limited diversity of symbols and rules, coupled with closed-ended answers. This paper proposes a method to leverage these shortcomings as useful features to a construct dynamic, counterfactual benchmark, which can be used to both reveal overfitting and measure true reasoning. We demonstrate this via MatheMagic, which generates math test instances with the interpretations of numbers and operators altered, yet has automatically verifiable answers. Test instances are randomly seeded and constructed at test time to evaluate a model's induction or deduction capability, offering stability, extensibility, comparability, and robustness to overfitting. Our experiments find that models solve deduction more easily than induction, but they revert to standard math. Further analysis reveals that math-adapted models fail to exhibit a general "skill" of reasoning, and fine-tuning on induction tasks generalizes poorly.