Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring
This work addresses the problem of reducing teacher workload and bias in grading for Austrian education, but it is incremental as it applies existing LLMs to a new dataset with limited success.
This paper tackled the problem of using large language models (LLMs) for automated essay scoring of Austrian A-level German essays, finding that the models achieved only 40.6% agreement with human raters on rubric sub-dimensions and 32.8% on final grades, indicating insufficient accuracy for real-world use.
Automated Essay Scoring (AES) has been explored for decades with the goal to support teachers by reducing grading workload and mitigating subjective biases. While early systems relied on handcrafted features and statistical models, recent advances in Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility. This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts, with a particular focus on rubric-based evaluation. A dataset of 101 anonymised student exams across three text types was processed and evaluated. Four LLMs, DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b, were evaluated with different contexts and prompting strategies. The LLMs were able to reach a maximum of 40.6% agreement with the human rater in the rubric-provided sub-dimensions, and only 32.8% of final grades matched the ones given by a human expert. The results indicate that even though smaller models are able to use standardised rubrics for German essay grading, they are not accurate enough to be used in a real-world grading environment.