LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages
This addresses the problem of assessing true mathematical reasoning in LLMs for AI researchers, though it is incremental as it builds on existing evaluation methods.
The paper tackled the challenge of evaluating LLMs' mathematical modeling abilities by proposing a process-oriented framework and introducing the Mamo benchmark with 1,209 questions, showing that existing LLMs struggle with complex tasks, with larger models performing better but open-source ones falling short in harder problems.
Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, yet their proficiency in mathematical reasoning remains a key challenge. Addressing the gap between natural and mathematical language requires advanced reasoning capabilities, approaching those of Artificial General Intelligence (AGI). However, the evaluation remains challenging, as perfectly representing reality is inherently elusive, and traditional methods like manual or direct comparison of mathematical statements (Ramamonjison et al., 2023) are insufficient for assessing true modeling ability. We propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. Introducing Mamo, a benchmark with 1,209 questions covering ordinary differential equations, linear programming, and mixed-integer linear programming, we enable automatic evaluation of modeling accuracy. The results show that existing LLMs struggle with complex mathematical modeling tasks, with larger models demonstrating superior performance, while open-source models remain competitive in simpler cases but still fall short of proprietary models in more challenging problems.