A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters
This work addresses the problem of memory and time consumption in DNN ensembles for practitioners, offering an incremental improvement by optimizing training and inference phases.
The paper tackles the challenge of building efficient ensembles of deep neural networks by proposing a new AutoML workflow that creates a library of diverse models and selects ensembles balancing accuracy and computational cost, demonstrating robust results on two datasets with efficient GPU cluster usage.
Automated Machine Learning with ensembling (or AutoML with ensembling) seeks to automatically build ensembles of Deep Neural Networks (DNNs) to achieve qualitative predictions. Ensemble of DNNs are well known to avoid over-fitting but they are memory and time consuming approaches. Therefore, an ideal AutoML would produce in one single run time different ensembles regarding accuracy and inference speed. While previous works on AutoML focus to search for the best model to maximize its generalization ability, we rather propose a new AutoML to build a larger library of accurate and diverse individual models to then construct ensembles. First, our extensive benchmarks show asynchronous Hyperband is an efficient and robust way to build a large number of diverse models to combine them. Then, a new ensemble selection method based on a multi-objective greedy algorithm is proposed to generate accurate ensembles by controlling their computing cost. Finally, we propose a novel algorithm to optimize the inference of the DNNs ensemble in a GPU cluster based on allocation optimization. The produced AutoML with ensemble method shows robust results on two datasets using efficiently GPU clusters during both the training phase and the inference phase.