Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?
This addresses a practical problem for deep learning practitioners by optimizing model performance within memory constraints, though it is incremental as it builds on existing ensemble and network width techniques.
The paper investigates whether a single wide network or an ensemble of thinner networks performs better under a fixed memory budget, finding that ensembles often outperform, such as achieving 82.52% vs. 80.6% accuracy on CIFAR-100 with WideResNet.
One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number of parameters is to increase the size of the network, e.g. width, or to train a deep ensemble; both approaches improve the performance in practice. In this work, we consider a fixed memory budget setting, and investigate, what is more effective: to train a single wide network, or to perform a memory split -- to train an ensemble of several thinner networks, with the same total number of parameters? We find that, for large enough budgets, the number of networks in the ensemble, corresponding to the optimal memory split, is usually larger than one. Interestingly, this effect holds for the commonly used sizes of the standard architectures. For example, one WideResNet-28-10 achieves significantly worse test accuracy on CIFAR-100 than an ensemble of sixteen thinner WideResNets: 80.6% and 82.52% correspondingly. We call the described effect the Memory Split Advantage and show that it holds for a variety of datasets and model architectures.