Empirical Analysis of the Effect of Context in the Task of Automated Essay Scoring in Transformer-Based Models
This work addresses the need for more effective AES in educational automation, offering an incremental improvement through contextual augmentation that is adaptable to existing models.
The study tackled the problem of transformer-based models underperforming in Automated Essay Scoring (AES) by enriching them with contextual factors, achieving a mean Quadratic Weighted Kappa score of 0.823 across the dataset and 0.8697 on individual sets, surpassing prior transformer models but still underperforming by an average of 3.83% compared to the state-of-the-art deep learning model.
Automated Essay Scoring (AES) has emerged to prominence in response to the growing demand for educational automation. Providing an objective and cost-effective solution, AES standardises the assessment of extended responses. Although substantial research has been conducted in this domain, recent investigations reveal that alternative deep-learning architectures outperform transformer-based models. Despite the successful dominance in the performance of the transformer architectures across various other tasks, this discrepancy has prompted a need to enrich transformer-based AES models through contextual enrichment. This study delves into diverse contextual factors using the ASAP-AES dataset, analysing their impact on transformer-based model performance. Our most effective model, augmented with multiple contextual dimensions, achieves a mean Quadratic Weighted Kappa score of 0.823 across the entire essay dataset and 0.8697 when trained on individual essay sets. Evidently surpassing prior transformer-based models, this augmented approach only underperforms relative to the state-of-the-art deep learning model trained essay-set-wise by an average of 3.83\% while exhibiting superior performance in three of the eight sets. Importantly, this enhancement is orthogonal to architecture-based advancements and seamlessly adaptable to any AES model. Consequently, this contextual augmentation methodology presents a versatile technique for refining AES capabilities, contributing to automated grading and evaluation evolution in educational settings.