MCAL: Minimum Cost Human-Machine Active Labeling
This work addresses the problem of expensive ground-truth generation for data annotation, offering a cost-effective solution for machine learning practitioners, though it is incremental in improving existing hybrid labeling strategies.
The paper tackles the high cost of human annotation by proposing a hybrid human-machine labeling approach that iteratively decides which samples to label by humans versus a trained classifier, achieving up to 6x lower overall cost compared to fully human-labeled datasets.
Today, ground-truth generation uses data sets annotated by cloud-based annotation services. These services rely on human annotation, which can be prohibitively expensive. In this paper, we consider the problem of hybrid human-machine labeling, which trains a classifier to accurately auto-label part of the data set. However, training the classifier can be expensive too. We propose an iterative approach that minimizes total overall cost by, at each step, jointly determining which samples to label using humans and which to label using the trained classifier. We validate our approach on well known public data sets such as Fashion-MNIST, CIFAR-10, CIFAR-100, and ImageNet. In some cases, our approach has 6x lower overall cost relative to human labeling the entire data set, and is always cheaper than the cheapest competing strategy.