Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model
This addresses the problem of interpretable and actionable feedback for students in education, though it is incremental as it builds on existing ASAG methods with domain adaptation.
The study tackled automated short answer grading by developing a model that provides analytic and holistic scores using one-shot prompting and text similarity scoring, achieving an accuracy of 0.67 and quadratic weighted kappa of 0.71 on a public dataset.
In this study, we developed an automated short answer grading (ASAG) model that provided both analytic scores and final holistic scores. Short answer items typically consist of multiple sub-questions, and providing an analytic score and the text span relevant to each sub-question can increase the interpretability of the automated scores. Furthermore, they can be used to generate actionable feedback for students. Despite these advantages, most studies have focused on predicting only holistic scores due to the difficulty in constructing dataset with manual annotations. To address this difficulty, we used large language model (LLM)-based one-shot prompting and a text similarity scoring model with domain adaptation using small manually annotated dataset. The accuracy and quadratic weighted kappa of our model were 0.67 and 0.71 on a subset of the publicly available ASAG dataset. The model achieved a substantial improvement over the majority baseline.