Evaluating LLM-Generated Code: A Benchmark and Developer Study
For researchers and practitioners selecting LLMs for code generation, this work provides a more comprehensive evaluation framework, though it is incremental as it combines existing techniques.
The paper introduces a three-fold evaluation methodology for LLM-generated code that goes beyond correctness to include code quality and developer opinions. Using this methodology, they evaluated GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4, finding that developer reviews provide insights into production-readiness not captured by standard benchmarks.
Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers' opinions on generated code samples gathered through a structured code-review process. The proposed methodology's usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.