Interactive Learning from Multiple Noisy Labels
This work addresses the challenge of improving learning efficiency in noisy annotation settings, but it appears incremental as it builds on existing interactive learning frameworks with a specific focus on disagreement metrics.
The paper tackles the problem of interactive learning from multiple noisy labels by using annotator disagreement to quantify example meaningfulness, and demonstrates its usefulness in parameter estimation for latent variable classification models with experimental analyses on synthetic and benchmark datasets.
Interactive learning is a process in which a machine learning algorithm is provided with meaningful, well-chosen examples as opposed to randomly chosen examples typical in standard supervised learning. In this paper, we propose a new method for interactive learning from multiple noisy labels where we exploit the disagreement among annotators to quantify the easiness (or meaningfulness) of an example. We demonstrate the usefulness of this method in estimating the parameters of a latent variable classification model, and conduct experimental analyses on a range of synthetic and benchmark datasets. Furthermore, we theoretically analyze the performance of perceptron in this interactive learning framework.