Neural Active Learning on Heteroskedastic Distributions
This addresses a critical issue for practitioners using active learning in real-world scenarios with varying data quality, though it is incremental in improving specific algorithm robustness.
The paper tackled the problem of active learning algorithms failing catastrophically on heteroskedastic distributions by preferring noisy data, and proposed a fine-tuning-based approach and a new algorithm with a model difference scoring function that outperforms existing techniques on such datasets.
Models that can actively seek out the best quality training data hold the promise of more accurate, adaptable, and efficient machine learning. Active learning techniques often tend to prefer examples that are the most difficult to classify. While this works well on homogeneous datasets, we find that it can lead to catastrophic failures when performed on multiple distributions with different degrees of label noise or heteroskedasticity. These active learning algorithms strongly prefer to draw from the distribution with more noise, even if their examples have no informative structure (such as solid color images with random labels). To this end, we demonstrate the catastrophic failure of these active learning algorithms on heteroskedastic distributions and propose a fine-tuning-based approach to mitigate these failures. Further, we propose a new algorithm that incorporates a model difference scoring function for each data point to filter out the noisy examples and sample clean examples that maximize accuracy, outperforming the existing active learning techniques on the heteroskedastic datasets. We hope these observations and techniques are immediately helpful to practitioners and can help to challenge common assumptions in the design of active learning algorithms.