Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
This addresses the need for more reliable evaluation methods in AI to prevent overestimation of reasoning abilities, though it is incremental as it builds on existing benchmarks.
The authors tackled the problem of robustly evaluating reasoning capabilities in language models by introducing functional variants of benchmarks, specifically rewriting part of the MATH benchmark into MATH(). They found a reasoning gap of 58.35% to 80.31% in state-of-the-art models, with models performing better on real-world tasks showing lower gaps.
We propose a framework for robust evaluation of reasoning capabilities of language models, using functional variants of benchmarks. Models that solve a reasoning test should exhibit no difference in performance over the static version of a problem compared to a snapshot of the functional variant. We have rewritten the relevant fragment of the MATH benchmark into its functional variant MATH(), with functionalization of other benchmarks to follow. When evaluating current state-of-the-art models over snapshots of MATH(), we find a reasoning gap -- the percentage difference between the static and functional accuracies. We find reasoning gaps from 58.35% to 80.31% among the state-of-the-art closed and open weights models that perform well on static benchmarks, with the caveat that the gaps are likely to be smaller with more sophisticated prompting strategies. Here we show that models which anecdotally have good reasoning performance over real-world tasks, have quantifiable lower gaps, motivating the open problem of building "gap 0" models. Code for evaluation and new evaluation datasets, three MATH() snapshots, are publicly available at https://github.com/consequentai/fneval/.