Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course
This addresses the practical application of LLMs for educational assessment, highlighting real-world challenges in a large-scale classroom setting, though it is incremental as it builds on existing LLM evaluation methods.
The study investigated using GPT-4 to automatically evaluate assignments in a university course with over 1,000 students, finding that students generally accepted it but noted issues like failure to follow instructions and vulnerability to manipulation for high scores.
Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. However, it is unclear whether these LLM-based evaluators can be applied in real-world classrooms to assess student assignments. This empirical report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students. Based on student responses, we find that LLM-based assignment evaluators are generally acceptable to students when students have free access to these LLM-based evaluators. However, students also noted that the LLM sometimes fails to adhere to the evaluation instructions. Additionally, we observe that students can easily manipulate the LLM-based evaluator to output specific strings, allowing them to achieve high scores without meeting the assignment rubric. Based on student feedback and our experience, we provide several recommendations for integrating LLM-based evaluators into future classrooms. Our observation also highlights potential directions for improving LLM-based evaluators, including their instruction-following ability and vulnerability to prompt hacking.