On the impact of predicate complexity in crowdsourced classification tasks
It addresses task design optimization for crowdsourcing platforms, which is an incremental improvement in a specific application area.
This paper investigates how different ways of formulating complex classification questions affect performance in crowdsourcing tasks, finding that predicate formulation significantly impacts classification accuracy across various domains and when combining human workers with machine learning classifiers.
This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.