CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback
This addresses the need for consistent and constructive feedback in programming education, though it appears incremental as it applies existing LLM techniques to a specific domain.
The authors tackled the problem of grading programming assignments by developing CodEv, an automated framework using Large Language Models (LLMs) with Chain of Thought prompting and ensembles, which achieved grading results comparable to human evaluators using smaller LLMs.
Grading programming assignments is crucial for guiding students to improve their programming skills and coding styles. This study presents an automated grading framework, CodEv, which leverages Large Language Models (LLMs) to provide consistent and constructive feedback. We incorporate Chain of Thought (CoT) prompting techniques to enhance the reasoning capabilities of LLMs and ensure that the grading is aligned with human evaluation. Our framework also integrates LLM ensembles to improve the accuracy and consistency of scores, along with agreement tests to deliver reliable feedback and code review comments. The results demonstrate that the framework can yield grading results comparable to human evaluators, by using smaller LLMs. Evaluation and consistency tests of the LLMs further validate our approach, confirming the reliability of the generated scores and feedback.