CrowdGrader: Crowdsourcing the Evaluation of Homework Assignments
This addresses the challenge of scalable grading in education, particularly for computer-science assignments, though it appears incremental as it builds on existing crowdsourcing methods.
The authors tackled the problem of evaluating homework assignments by developing CrowdGrader, a tool that uses crowdsourcing with a novel algorithm combining student-provided grades and a reputation system to compute consensus grades, showing better performance on synthetic data than alternatives and effective incentives in practice.
Crowdsourcing offers a practical method for ranking and scoring large amounts of items. To investigate the algorithms and incentives that can be used in crowdsourcing quality evaluations, we built CrowdGrader, a tool that lets students submit and collaboratively grade solutions to homework assignments. We present the algorithms and techniques used in CrowdGrader, and we describe our results and experience in using the tool for several computer-science assignments. CrowdGrader combines the student-provided grades into a consensus grade for each submission using a novel crowdsourcing algorithm that relies on a reputation system. The algorithm iterativerly refines inter-dependent estimates of the consensus grades, and of the grading accuracy of each student. On synthetic data, the algorithm performs better than alternatives not based on reputation. On our preliminary experimental data, the performance seems dependent on the nature of review errors, with errors that can be ascribed to the reviewer being more tractable than those arising from random external events. To provide an incentive for reviewers, the grade each student receives in an assignment is a combination of the consensus grade received by their submissions, and of a reviewing grade capturing their reviewing effort and accuracy. This incentive worked well in practice.