CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
This addresses the need for scalable and explainable code evaluation methods for researchers and practitioners in neural code generation, though it builds incrementally on existing LLM-as-Judge approaches.
The paper tackles the problem of evaluating code snippets in neural code generation by proposing CODE-DITING, a method that balances accuracy, efficiency, and explainability. The result shows that CODE-DITING 1.5B outperforms models with similar parameters and achieves performance typical of models 5 times larger, while CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B using only 1% of their parameters.
Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose CODE-DITING, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1671B to our CODE-DITING 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, CODE-DITING 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that CODEDITING is robust to preference leakage and can serve as a promising alternative for code evaluation.