Towards Accelerated Model Training via Bayesian Data Selection
This work addresses training efficiency issues for machine learning practitioners dealing with noisy or imbalanced datasets, though it is incremental as it builds on prior data selection principles.
The paper tackles the problem of mislabeled, duplicated, or biased data slowing down model training by proposing a Bayesian data selection method that uses zero-shot predictors, achieving similar predictive performance with significantly fewer training iterations on benchmarks like WebVision.
Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.