ASAG2024: A Combined Benchmark for Short Answer Grading
This work addresses the problem of assessing generalizability in automated grading for educators and researchers, though it is incremental as it builds on existing datasets and methods.
The paper tackled the lack of a comprehensive benchmark for Short Answer Grading (SAG) systems by introducing ASAG2024, which combines seven datasets, and found that LLM-based methods achieve new high scores but still fall short of human performance.
Open-ended questions test a more thorough understanding than closed-ended questions and are often a preferred assessment method. However, open-ended questions are tedious to grade and subject to personal bias. Therefore, there have been efforts to speed up the grading process through automation. Short Answer Grading (SAG) systems aim to automatically score students' answers. Despite growth in SAG methods and capabilities, there exists no comprehensive short-answer grading benchmark across different subjects, grading scales, and distributions. Thus, it is hard to assess the capabilities of current automated grading methods in terms of their generalizability. In this preliminary work, we introduce the combined ASAG2024 benchmark to facilitate the comparison of automated grading systems. Combining seven commonly used short-answer grading datasets in a common structure and grading scale. For our benchmark, we evaluate a set of recent SAG methods, revealing that while LLM-based approaches reach new high scores, they still are far from reaching human performance. This opens up avenues for future research on human-machine SAG systems.