Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring
This work addresses automated educational assessment for educators and institutions, representing an incremental advancement by integrating retrieval-augmented generation into existing GLM frameworks.
The study tackled automated short answer scoring by proposing a pipeline combining vector databases, transformer encoders, and generative language models, achieving significant improvements on the SemEval 2013 dataset compared to existing methods.
Automated Short Answer Scoring (ASAS) is a critical component in educational assessment. While traditional ASAS systems relied on rule-based algorithms or complex deep learning methods, recent advancements in Generative Language Models (GLMs) offer new opportunities for improvement. This study explores the application of GLMs to ASAS, leveraging their off-the-shelf capabilities and performance in various domains. We propose a novel pipeline that combines vector databases, transformer-based encoders, and GLMs to enhance short answer scoring accuracy. Our approach stores training responses in a vector database, retrieves semantically similar responses during inference, and employs a GLM to analyze these responses and determine appropriate scores. We further optimize the system through fine-tuned retrieval processes and prompt engineering. Evaluation on the SemEval 2013 dataset demonstrates a significant improvement on the SCIENTSBANK 3-way and 2-way tasks compared to existing methods, highlighting the potential of GLMs in advancing ASAS technology.