Layer Ensembles
This addresses uncertainty estimation for deep learning practitioners by offering a more efficient and higher-quality alternative to Deep Ensembles, though it is incremental as it builds on existing ensemble methods.
The paper tackles uncertainty estimation in neural networks by proposing Layer Ensembles, which use independent categorical distributions per layer to generate more samples than Deep Ensembles, achieving up to 19x speedup and reduced memory usage while improving uncertainty quality.
Deep Ensembles, as a type of Bayesian Neural Networks, can be used to estimate uncertainty on the prediction of multiple neural networks by collecting votes from each network and computing the difference in those predictions. In this paper, we introduce a method for uncertainty estimation that considers a set of independent categorical distributions for each layer of the network, giving many more possible samples with overlapped layers than in the regular Deep Ensembles. We further introduce an optimized inference procedure that reuses common layer outputs, achieving up to 19x speed up and reducing memory usage quadratically. We also show that the method can be further improved by ranking samples, resulting in models that require less memory and time to run while achieving higher uncertainty quality than Deep Ensembles.