Automated Text Scoring in the Age of Generative AI for the GPU-poor
This addresses transparency and efficiency issues in automated text scoring for resource-limited users, though it is incremental.
The study tackled automated text scoring using open-source generative language models on consumer-grade hardware, finding that fine-tuned models achieve adequate but not state-of-the-art performance, with initial steps showing promise for generating feedback.
Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.