CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement
This work addresses the challenge of quality assessment in crowdsourcing for domains with inherent data ambiguity, though it appears incremental as an extension of prior metrics.
The paper tackles the problem of ambiguous data and multiple perspectives in crowdsourcing by introducing CrowdTruth metrics that interpret inter-annotator disagreement, modeling the interdependency between workers, input data, and annotations to capture ambiguity in each component.
Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system -- worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core .