MAGIC: Multi-Agent Argumentation and Grammar Integrated Critiquer
This addresses the need for better automated essay feedback systems for college-level education, offering an incremental improvement over existing methods by integrating specialized agents and a new dataset.
The paper tackles the problem of automated essay scoring and feedback at the college level, where existing systems focus on scoring accuracy and lower-grade writing. It introduces MAGIC, a multi-agent framework that achieves substantial to near-perfect scoring agreement with humans on GRE essays and provides high-quality, interpretable feedback.
Automated Essay Scoring (AES) and Automatic Essay Feedback (AEF) systems aim to reduce the workload of human raters in educational assessment. However, most existing systems prioritize numerical scoring accuracy over feedback quality and are primarily evaluated on pre-secondary school level writing. This paper presents Multi-Agent Argumentation and Grammar Integrated Critiquer (MAGIC), a framework using five specialized agents to evaluate prompt adherence, persuasiveness, organization, vocabulary, and grammar for both holistic scoring and detailed feedback generation. To support evaluation at the college level, we collated a dataset of Graduate Record Examination (GRE) practice essays with expert-evaluated scores and feedback. MAGIC achieves substantial to near-perfect scoring agreement with humans on the GRE data, outperforming baseline LLM models while providing enhanced interpretability through its multi-agent approach. We also compare MAGIC's feedback generation capabilities against ground truth human feedback and baseline models, finding that MAGIC achieves strong feedback quality and naturalness.