Unlearn What You Have Learned: Adaptive Crowd Teaching with Exponentially Decayed Memory Learners
This addresses the problem of efficiently supervising crowd labeling for data mining applications, though it appears incremental as it builds on existing machine teaching concepts in crowdsourcing.
The paper tackles adaptive crowd teaching by proposing the JEDI framework, which constructs personalized optimal teaching sets for crowdsourcing workers, assuming learners have exponentially decayed memory, and validates its effectiveness against state-of-the-art techniques on multiple datasets.
With the increasing demand for large amount of labeled data, crowdsourcing has been used in many large-scale data mining applications. However, most existing works in crowdsourcing mainly focus on label inference and incentive design. In this paper, we address a different problem of adaptive crowd teaching, which is a sub-area of machine teaching in the context of crowdsourcing. Compared with machines, human beings are extremely good at learning a specific target concept (e.g., classifying the images into given categories) and they can also easily transfer the learned concepts into similar learning tasks. Therefore, a more effective way of utilizing crowdsourcing is by supervising the crowd to label in the form of teaching. In order to perform the teaching and expertise estimation simultaneously, we propose an adaptive teaching framework named JEDI to construct the personalized optimal teaching set for the crowdsourcing workers. In JEDI teaching, the teacher assumes that each learner has an exponentially decayed memory. Furthermore, it ensures comprehensiveness in the learning process by carefully balancing teaching diversity and learner's accurate learning in terms of teaching usefulness. Finally, we validate the effectiveness and efficacy of JEDI teaching in comparison with the state-of-the-art techniques on multiple data sets with both synthetic learners and real crowdsourcing workers.