A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions
This work provides a comprehensive resource for researchers and practitioners in operations research and AI by surveying automation methods and improving benchmark quality, though it is incremental as a survey and analysis.
This survey reviews recent progress in using large language models (LLMs) to automate optimization modeling, which traditionally requires operations research expertise, and includes an analysis revealing high error rates in benchmark datasets, leading to cleaned datasets and a new leaderboard.
By virtue of its great utility in solving real-world problems, optimization modeling has been widely employed for optimal decision-making across various sectors, but it requires substantial expertise from operations research professionals. With the advent of large language models (LLMs), new opportunities have emerged to automate the procedure of mathematical modeling. This survey presents a comprehensive and timely review of recent advancements that cover the entire technical stack, including data synthesis and fine-tuning for the base model, inference frameworks, benchmark datasets, and performance evaluation. In addition, we conducted an in-depth analysis on the quality of benchmark datasets, which was found to have a surprisingly high error rate. We cleaned the datasets and constructed a new leaderboard with fair performance evaluation in terms of base LLM model and datasets. We also build an online portal that integrates resources of cleaned datasets, code and paper repository to benefit the community. Finally, we identify limitations in current methodologies and outline future research opportunities.