FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback
This addresses the issue of unreliable feedback propagation in automated essay scoring for educational applications, though it is incremental as it builds on existing LLM-based evaluation methods.
The paper tackles the problem of noise in LLM-generated essay feedback used for training automated scoring models by proposing FeedEval, a framework that evaluates feedback along pedagogically grounded dimensions, resulting in improved scoring performance and more effective essay revisions as shown on the ASAP++ benchmark.
Going beyond the prediction of numerical scores, recent research in automated essay scoring has increasingly emphasized the generation of high-quality feedback that provides justification and actionable guidance. To mitigate the high cost of expert annotation, prior work has commonly relied on LLM-generated feedback to train essay assessment models. However, such feedback is often incorporated without explicit quality validation, resulting in the propagation of noise in downstream applications. To address this limitation, we propose FeedEval, an LLM-based framework for evaluating LLM-generated essay feedback along three pedagogically grounded dimensions: specificity, helpfulness, and validity. FeedEval employs dimension-specialized LLM evaluators trained on datasets curated in this study to assess multiple feedback candidates and select high-quality feedback for downstream use. Experiments on the ASAP++ benchmark show that FeedEval closely aligns with human expert judgments and that essay scoring models trained with FeedEval-filtered high-quality feedback achieve superior scoring performance. Furthermore, revision experiments using small LLMs show that the high-quality feedback identified by FeedEval leads to more effective essay revisions. We will release our code and curated datasets upon accepted.