Comprehensive Benchmarking of Entropy and Margin Based Scoring Metrics for Data Selection
This work addresses data selection challenges for industry-scale applications, particularly in low-resource languages, but is incremental as it builds on existing metrics like entropy and EL2N.
The paper tackled the problem of data selection for training supervised machine learning models in low-resource language settings, demonstrating that using entropy and EL2N scores to curate datasets from weak signal labeled data resulted in a 2% decrease in semantic error rate and 4%-7% decrease in domain classification error rate compared to random selection.
While data selection methods have been studied extensively in active learning, data pruning, and data augmentation settings, there is little evidence for the efficacy of these methods in industry scale settings, particularly in low-resource languages. Our work presents ways of assessing prospective training examples in those settings for their "usefulness" or "difficulty". We also demonstrate how these measures can be used in selecting important examples for training supervised machine learning models. We primarily experiment with entropy and Error L2-Norm (EL2N) scores. We use these metrics to curate high quality datasets from a large pool of \textit{Weak Signal Labeled} data, which assigns no-defect high confidence hypotheses during inference as ground truth labels. We then conduct training data augmentation experiments using these de-identified datasets and demonstrate that score-based selection can result in a 2% decrease in semantic error rate and 4%-7% decrease in domain classification error rate when compared to the baseline technique of random selection.