Bayesian Optimization Meets Self-Distillation
This work addresses hyperparameter optimization and model training inefficiencies for machine learning practitioners, representing an incremental improvement by integrating existing methods.
The paper tackles the problem of inefficient knowledge transfer in Bayesian optimization and self-distillation by proposing the BOSS framework, which combines both to leverage knowledge from all training trials, achieving significantly better performance across tasks like image classification and medical image analysis.
Bayesian optimization (BO) has contributed greatly to improving model performance by suggesting promising hyperparameter configurations iteratively based on observations from multiple training trials. However, only partial knowledge (i.e., the measured performances of trained models and their hyperparameter configurations) from previous trials is transferred. On the other hand, Self-Distillation (SD) only transfers partial knowledge learned by the task model itself. To fully leverage the various knowledge gained from all training trials, we propose the BOSS framework, which combines BO and SD. BOSS suggests promising hyperparameter configurations through BO and carefully selects pre-trained models from previous trials for SD, which are otherwise abandoned in the conventional BO process. BOSS achieves significantly better performance than both BO and SD in a wide range of tasks including general image classification, learning with noisy labels, semi-supervised learning, and medical image analysis tasks.