Calibrating Generative AI to Produce Realistic Essays for Data Augmentation
This work addresses data scarcity in educational assessment tools, but it is incremental as it compares existing prompting methods rather than introducing new techniques.
This study tackled the problem of limited training data for automated scoring engines by evaluating three large language model prompting approaches for generating realistic student essays for data augmentation. The results showed that the predict next strategy achieved the highest human rater agreement on simulated essay scores and produced the most realistic text, while predict next and sentence strategies best preserved the original essay quality.
Data augmentation can mitigate limited training data in machine-learning automated scoring engines for constructed response items. This study seeks to determine how well three approaches to large language model prompting produce essays that preserve the writing quality of the original essays and produce realistic text for augmenting ASE training datasets. We created simulated versions of student essays, and human raters assigned scores to them and rated the realism of the generated text. The results of the study indicate that the predict next prompting strategy produces the highest level of agreement between human raters regarding simulated essay scores, predict next and sentence strategies best preserve the rated quality of the original essay in the simulated essays, and predict next and 25 examples strategies produce the most realistic text as judged by human raters.