ChatGPT as a Solver and Grader of Programming Exams written in Spanish
This work addresses the problem of evaluating AI assistance in education for teachers and students, but it is incremental as it builds on existing research on LLMs in educational contexts.
The study assessed ChatGPT's ability to solve and grade Spanish-language programming exams from a Computer Science degree, finding it effective only for simple coding tasks and ineffective for complex problems or evaluating others' solutions.
Evaluating the capabilities of Large Language Models (LLMs) to assist teachers and students in educational tasks is receiving increasing attention. In this paper, we assess ChatGPT's capacities to solve and grade real programming exams, from an accredited BSc degree in Computer Science, written in Spanish. Our findings suggest that this AI model is only effective for solving simple coding tasks. Its proficiency in tackling complex problems or evaluating solutions authored by others are far from effective. As part of this research, we also release a new corpus of programming tasks and the corresponding prompts for solving the problems or grading the solutions. This resource can be further exploited by other research teams.