HyperGAN: A Generative Model for Diverse, Performant Neural Networks
This addresses uncertainty estimation for neural networks, which is crucial for safety-critical applications, though it appears incremental as it builds on existing generative and ensemble methods.
The paper tackles the problem of neural networks being overconfident on out-of-distribution data by introducing HyperGAN, a generative model that learns a distribution of neural network parameters, enabling the creation of diverse ensembles. It shows competitive performance on MNIST and CIFAR-10 datasets and provides better uncertainty estimates than standard ensembles.
Standard neural networks are often overconfident when presented with data outside the training distribution. We introduce HyperGAN, a new generative model for learning a distribution of neural network parameters. HyperGAN does not require restrictive assumptions on priors, and networks sampled from it can be used to quickly create very large and diverse ensembles. HyperGAN employs a novel mixer to project prior samples to a latent space with correlated dimensions, and samples from the latent space are then used to generate weights for each layer of a deep neural network. We show that HyperGAN can learn to generate parameters which label the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty estimates than standard ensembles by evaluating on out of distribution data as well as adversarial examples.