Self-Distribution Distillation: Efficient Uncertainty Estimation
This addresses the need for efficient uncertainty estimation in safety-critical domains, offering a novel training method that reduces resource requirements compared to ensembles.
The paper tackles the problem of high computational cost in uncertainty estimation for deep learning by proposing Self-Distribution Distillation (S2D), which trains a single model to efficiently estimate uncertainties, outperforming standard models and Monte-Carlo dropout on CIFAR-100 and even deep ensembles in out-of-distribution detection on datasets like LSUN.
Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model's prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently train a single model that can estimate uncertainties. Furthermore it is possible to build ensembles of these models and apply hierarchical ensemble distillation approaches. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a standard deep ensemble can be outperformed using S2D based ensembles and novel distilled models.