Distributional Ground Truth: Non-Redundant Crowdsourcing Data Quality Control in UI Labeling Tasks
This work addresses the problem of reducing ancillary work effort and expenses in crowdsourcing data quality control for HCI and ML practitioners, particularly in UI labeling tasks, by offering a non-redundant quality prediction method.
This paper introduces a non-redundant method for predicting crowdworker output quality in UI labeling tasks, using a distributional ground truth approach based on the Kolmogorov-Smirnov test. The method achieved R2s of over 0.8 with a trusted set size of 17-27% of UIs, outperforming a baseline model based on mean Time-on-Task.
HCI increasingly employs Machine Learning and Image Recognition, in particular for visual analysis of user interfaces (UIs). A popular way for obtaining human-labeled training data is Crowdsourcing, typically using the quality control methods ground truth and majority consensus, which necessitate redundancy in the outcome. In our paper we propose a non-redundant method for prediction of crowdworkers' output quality in web UI labeling tasks, based on homogeneity of distributions assessed with two-sample Kolmogorov-Smirnov test. Using a dataset of about 500 screenshots with over 74,000 UI elements located and classified by 11 trusted labelers and 298 Amazon Mechanical Turk crowdworkers, we demonstrate the advantage of our approach over the baseline model based on mean Time-on-Task. Exploring different dataset partitions, we show that with the trusted set size of 17-27% UIs our "distributional ground truth" model can achieve R2s of over 0.8 and help to obviate the ancillary work effort and expenses.