Using ChatGPT to Score Essays and Short-Form Constructed Responses
This addresses the problem of automating essay scoring for educational assessments, but it is incremental as it builds on existing machine scoring methods with a new tool.
The study evaluated ChatGPT's ability to score essays and short-form responses, comparing it to human raters using quadratic weighted kappa metrics, and found that while its gradient boost model sometimes approached human accuracy, overall performance was inconsistent and often lower.
This study aimed to determine if ChatGPT's large language models could match the scoring accuracy of human and machine scores from the ASAP competition. The investigation focused on various prediction models, including linear regression, random forest, gradient boost, and boost. ChatGPT's performance was evaluated against human raters using quadratic weighted kappa (QWK) metrics. Results indicated that while ChatGPT's gradient boost model achieved QWKs close to human raters for some data sets, its overall performance was inconsistent and often lower than human scores. The study highlighted the need for further refinement, particularly in handling biases and ensuring scoring fairness. Despite these challenges, ChatGPT demonstrated potential for scoring efficiency, especially with domain-specific fine-tuning. The study concludes that ChatGPT can complement human scoring but requires additional development to be reliable for high-stakes assessments. Future research should improve model accuracy, address ethical considerations, and explore hybrid models combining ChatGPT with empirical methods.