Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments
This work addresses the challenge of assessing LLMs' generalizability in mathematical problem-solving for researchers and educators, though it is incremental as it builds on existing evaluation methods with a new dataset.
The paper tackled the problem of evaluating large language models' (LLMs) mathematical reasoning abilities in combinatorics by comparing models like GPT-4 against humans using a new dataset, Combi-Puzzles, with 125 variants; they found that GPT-4 outperformed other models and humans in mathematical variations, with modifications affecting LLM performance but not human performance.
In this paper we look at the ability of recent large language models (LLMs) at solving mathematical problems in combinatorics. We compare models LLaMA-2, LLaMA-3.1, GPT-4, and Mixtral against each other and against human pupils and undergraduates with prior experience in mathematical olympiads. To facilitate these comparisons we introduce the Combi-Puzzles dataset, which contains 125 problem variants based on 25 combinatorial reasoning problems. Each problem is presented in one of five distinct forms, created by systematically manipulating the problem statements through adversarial additions, numeric parameter changes, and linguistic obfuscation. Our variations preserve the mathematical core and are designed to measure the generalisability of LLM problem-solving abilities, while also increasing confidence that problems are submitted to LLMs in forms that have not been seen as training instances. We found that a model based on GPT-4 outperformed all other models in producing correct responses, and performed significantly better in the mathematical variation of the problems than humans. We also found that modifications to problem statements significantly impact the LLM's performance, while human performance remains unaffected.