Holistic Evaluation of State-of-the-Art LLMs for Code Generation
This provides actionable recommendations for developers and practitioners on deploying LLMs for reliable code generation in real-world software development tasks, though it is incremental as it focuses on benchmarking existing models.
This study evaluated six state-of-the-art large language models for code generation on 944 real-world LeetCode problems, finding that DeepSeek-R1 and GPT-4.1 consistently outperformed others in correctness, efficiency, and robustness.
This study presents a comprehensive empirical evaluation of six state-of-the-art large language models (LLMs) for code generation, including both general-purpose and code-specialized models. Using a dataset of 944 real-world LeetCode problems across five programming languages, we assess model performance using rigorous metrics: compile-time errors, runtime errors, functional failures, and algorithmic suboptimalities. The results reveal significant performance variations, with DeepSeek-R1 and GPT-4.1 consistently outperform others in terms of correctness, efficiency, and robustness. Through detailed case studies, we identify common failure scenarios such as syntax errors, logical flaws, and suboptimal algorithms, highlighting the critical role of prompt engineering and human oversight in improving results. Based on these findings, we provide actionable recommendations for developers and practitioners, emphasizing that successful LLM deployment depends on careful model selection, effective prompt design, and context-aware usage to ensure reliable code generation in real-world software development tasks.