Assessing instructor-AI cooperation for grading essay-type questions in an introductory sociology course
This addresses grading efficiency and fairness in higher education sociology courses, but is incremental as it positions AI as a complementary tool rather than a replacement.
This study evaluated GPT models for transcribing and grading handwritten sociology exam essays, finding high similarity with human transcriptions (GPT-4o-mini performed best) and strong grading correlations when template answers were provided, though discrepancies remained.
This study explores the use of artificial intelligence (AI) as a complementary tool for grading essay-type questions in higher education, focusing on its consistency with human grading and potential to reduce biases. Using 70 handwritten exams from an introductory sociology course, we evaluated generative pre-trained transformers (GPT) models' performance in transcribing and scoring students' responses. GPT models were tested under various settings for both transcription and grading tasks. Results show high similarity between human and GPT transcriptions, with GPT-4o-mini outperforming GPT-4o in accuracy. For grading, GPT demonstrated strong correlations with the human grader scores, especially when template answers were provided. However, discrepancies remained, highlighting GPT's role as a "second grader" to flag inconsistencies for assessment reviewing rather than fully replace human evaluation. This study contributes to the growing literature on AI in education, demonstrating its potential to enhance fairness and efficiency in grading essay-type questions.