Deep Neural Network Ensembles
This work addresses interpretability and overfitting issues for deep learning practitioners, offering a novel ensemble method with theoretical backing, though it is incremental in building on existing regularization and ensemble techniques.
The paper tackled the problems of interpretability and overfitting in deep neural networks by analyzing hidden space paths to extract decision-making features and proposing an ensemble algorithm with test accuracy guarantees, achieving state-of-the-art results on CIFAR-10 with Wide-ResNets and improving test accuracy across all applied models.
Current deep neural networks suffer from two problems; first, they are hard to interpret, and second, they suffer from overfitting. There have been many attempts to define interpretability in neural networks, but they typically lack causality or generality. A myriad of regularization techniques have been developed to prevent overfitting, and this has driven deep learning to become the hot topic it is today; however, while most regularization techniques are justified empirically and even intuitively, there is not much underlying theory. This paper argues that to extract the features used in neural networks to make decisions, it's important to look at the paths between clusters existing in the hidden spaces of neural networks. These features are of particular interest because they reflect the true decision making process of the neural network. This analysis is then furthered to present an ensemble algorithm for arbitrary neural networks which has guarantees for test accuracy. Finally, a discussion detailing the aforementioned guarantees is introduced and the implications to neural networks, including an intuitive explanation for all current regularization methods, are presented. The ensemble algorithm has generated state-of-the-art results for Wide-ResNets on CIFAR-10 (top 5 for all models) and has improved test accuracy for all models it has been applied to.