EnsembleNet: End-to-End Optimization of Multi-headed Models
This work addresses the inefficiency of traditional ensemble methods for machine learning practitioners by enabling end-to-end optimization in a single stage.
The paper tackles the problem of inefficient ensemble training by proposing a simpler co-distillation architecture that converts a single neural network into a multi-headed model, achieving better performance and smaller size on datasets like ImageNet, YouTube-8M, and Kinetics.
Ensembling is a universally useful approach to boost the performance of machine learning models. However, individual models in an ensemble were traditionally trained independently in separate stages without information access about the overall ensemble. Many co-distillation approaches were proposed in order to treat model ensembling as first-class citizens. In this paper, we reveal a deeper connection between ensembling and distillation, and come up with a simpler yet more effective co-distillation architecture. On large-scale datasets including ImageNet, YouTube-8M, and Kinetics, we demonstrate a general procedure that can convert a single deep neural network to a multi-headed model that has not only a smaller size but also better performance. The model can be optimized end-to-end with our proposed co-distillation loss in a single stage without human intervention.