Evaluating Crowdsourcing Participants in the Absence of Ground-Truth

This addresses the challenge of ensuring data quality in crowdsourcing for machine learning, but it appears incremental as it builds on existing work in annotator reliability.

The paper tackles the problem of identifying adversarial or unreliable annotators in supervised or semi-supervised learning scenarios with multiple annotators, but the abstract does not provide specific results or concrete numbers.