Crowdsourcing in the Absence of Ground Truth -- A Case Study
This addresses the problem of eliciting reliable insights from subjective crowdsourced data for researchers in AI and HCI, though it is incremental as it builds on existing methods for aggregation and comparison.
The paper tackled the challenge of aggregating subjective crowdsourced evaluations for estimating emotions in distressed individuals, finding that simple voting consensus is as effective as optimal aggregation and a machine learning algorithm matches human reliability in assessing dominant emotions.
Crowdsourcing information constitutes an important aspect of human-in-the-loop learning for researchers across multiple disciplines such as AI, HCI, and social science. While using crowdsourced data for subjective tasks is not new, eliciting useful insights from such data remains challenging due to a variety of factors such as difficulty of the task, personal prejudices of the human evaluators, lack of question clarity, etc. In this paper, we consider one such subjective evaluation task, namely that of estimating experienced emotions of distressed individuals who are conversing with a human listener in an online coaching platform. We explore strategies to aggregate the evaluators choices, and show that a simple voting consensus is as effective as an optimum aggregation method for the task considered. Intrigued by how an objective assessment would compare to the subjective evaluation of evaluators, we also designed a machine learning algorithm to perform the same task. Interestingly, we observed a machine learning algorithm that is not explicitly modeled to characterize evaluators' subjectivity is as reliable as the human evaluation in terms of assessing the most dominant experienced emotions.