Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions
This work addresses the need for more rigorous evaluation of LLMs' reasoning capabilities for researchers and developers, though it is incremental as it builds on existing benchmarks with new perturbations.
The paper tackled the problem of assessing the true depth and robustness of Large Language Models (LLMs) in reasoning tasks by introducing ontology-guided perturbations to math and coding questions, resulting in a significant performance drop across models, indicating a lack of robust problem-solving skills.
Recent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMORE and HUMANEVAL-CORE, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks. Through comprehensive evaluations of both closed-source and open-source LLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology. We open-source the datasets and source codes at: https://github.com/declare-lab/LLM-ReasoningTest.