Embedded Ensembles: Infinite Width Limit and Operating Regimes
This work addresses memory efficiency for practitioners using neural network ensembles, but it is incremental as it builds on existing EE methods with theoretical and empirical analysis.
The paper tackles the problem of memory-efficient ensembling of neural networks by analyzing Embedded Ensembling (EE) methods like BatchEnsembles and Monte-Carlo dropout, deriving a wide network limit using Neural Tangent Kernel theory to identify independent and collective regimes, and confirming through experiments that in the independent regime, EE behaves like an ensemble of independent models.
A memory efficient approach to ensembling neural networks is to share most weights among the ensembled models by means of a single reference network. We refer to this strategy as Embedded Ensembling (EE); its particular examples are BatchEnsembles and Monte-Carlo dropout ensembles. In this paper we perform a systematic theoretical and empirical analysis of embedded ensembles with different number of models. Theoretically, we use a Neural-Tangent-Kernel-based approach to derive the wide network limit of the gradient descent dynamics. In this limit, we identify two ensemble regimes - independent and collective - depending on the architecture and initialization strategy of ensemble models. We prove that in the independent regime the embedded ensemble behaves as an ensemble of independent models. We confirm our theoretical prediction with a wide range of experiments with finite networks, and further study empirically various effects such as transition between the two regimes, scaling of ensemble performance with the network width and number of models, and dependence of performance on a number of architecture and hyperparameter choices.