Building Hierarchies of Concepts via Crowdsourcing
This addresses the need for scalable and cost-effective hierarchy creation in applications like navigation and organization, offering an incremental improvement over centralized expert methods.
The paper tackles the problem of building concept hierarchies by proposing a crowdsourcing system that captures uncertainty and actively selects questions using information gain, showing robustness to noise, efficiency, and cost-effectiveness in evaluations on simulated and real-world data.
Hierarchies of concepts are useful in many applications from navigation to organization of objects. Usually, a hierarchy is created in a centralized manner by employing a group of domain experts, a time-consuming and expensive process. The experts often design one single hierarchy to best explain the semantic relationships among the concepts, and ignore the natural uncertainty that may exist in the process. In this paper, we propose a crowdsourcing system to build a hierarchy and furthermore capture the underlying uncertainty. Our system maintains a distribution over possible hierarchies and actively selects questions to ask using an information gain criterion. We evaluate our methodology on simulated data and on a set of real world application domains. Experimental results show that our system is robust to noise, efficient in picking questions, cost-effective and builds high quality hierarchies.