How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling
This work addresses the challenge of assessing LLMs for complex real-world applications, highlighting critical deficiencies that hinder their practical use in domains like mathematical modeling.
The authors tackled the problem of evaluating LLMs' end-to-end problem-solving ability in real-world mathematical modeling competitions, revealing a comprehension-execution gap where LLMs perform well in early stages but fail in execution-oriented tasks, with errors persisting despite model scaling.
Large language models (LLMs) have achieved strong performance on reasoning benchmarks, yet their ability to solve real-world problems requiring end-to-end workflows remains unclear. Mathematical modeling competitions provide a stringent testbed for evaluating such end-to-end problem-solving capability. We propose a problem-oriented, stage-wise evaluation framework that assesses LLM performance across modeling stages using expert-verified criteria. We validate the framework's reliability by comparing automatic scores with independent human expert judgments on problems from the China Postgraduate Mathematical Contest in Modeling, demonstrating substantially stronger alignment than existing evaluation schemes. Using this framework, we reveal a comprehension-execution gap in state-of-the-art LLMs: while they perform well in early stages such as problem identification and formulation, they exhibit persistent deficiencies in execution-oriented stages including model solving, code implementation, and result analysis. These gaps persist even with increased model scale. We further trace these failures to insufficient specification, missing verification, and lack of validation, with errors propagating across stages without correction. Our findings suggest that bridging this gap requires approaches beyond model scaling, offering insights for applying LLMs to complex real-world problem solving.