Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queries
This work addresses the challenge of capturing nuanced human judgment in machine learning systems, offering a more information-efficient approach for tasks like sentiment and aesthetic classification, though it is incremental in improving existing human-in-the-loop paradigms.
The paper tackles the problem of inefficient human-in-the-loop learning by developing a framework that uses ranking and selection queries instead of just labels, resulting in a 57% reduction in learning time for a sentiment classification task compared to traditional methods.
Integrating human expertise into machine learning systems often reduces the role of experts to labeling oracles, a paradigm that limits the amount of information exchanged and fails to capture the nuances of human judgment. We address this challenge by developing a human-in-the-loop framework to learn binary classifiers with rich query types, consisting of item ranking and exemplar selection. We first introduce probabilistic human response models for these rich queries motivated by the relationship experimentally observed between the perceived implicit score of an item and its distance to the unknown classifier. Using these models, we then design active learning algorithms that leverage the rich queries to increase the information gained per interaction. We provide theoretical bounds on sample complexity and develop a tractable and computationally efficient variational approximation. Through experiments with simulated annotators derived from crowdsourced word-sentiment and image-aesthetic datasets, we demonstrate significant reductions on sample complexity. We further extend active learning strategies to select queries that maximize information rate, explicitly balancing informational value against annotation cost. This algorithm in the word sentiment classification task reduces learning time by more than 57\% compared to traditional label-only active learning.