Detecting Problem Statements in Peer Assessments
This work addresses the need for efficient quality control in educational peer assessments, but it is incremental as it applies existing methods to a specific dataset.
The paper tackled the problem of automating the detection of whether peer assessment review comments identify problems, using over 18,000 labeled comments. The best result was a Hierarchical Attention Network classifier achieving 93.1% accuracy, outperforming other neural and traditional models.
Effective peer assessment requires students to be attentive to the deficiencies in the work they rate. Thus, their reviews should identify problems. But what ways are there to check that they do? We attempt to automate the process of deciding whether a review comment detects a problem. We use over 18,000 review comments that were labeled by the reviewees as either detecting or not detecting a problem with the work. We deploy several traditional machine-learning models, as well as neural-network models using GloVe and BERT embeddings. We find that the best performer is the Hierarchical Attention Network classifier, followed by the Bidirectional Gated Recurrent Units (GRU) Attention and Capsule model with scores of 93.1% and 90.5% respectively. The best non-neural network model was the support vector machine with a score of 89.71%. This is followed by the Stochastic Gradient Descent model and the Logistic Regression model with 89.70% and 88.98%.