A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
This work addresses the problem of enhancing CNN performance and data efficiency for researchers and practitioners in computer vision, though it is incremental as it builds on existing CNN architectures.
The paper tackles the problem of improving generalization and training efficiency in convolutional neural networks (CNNs) by proposing a Bayesian generative model, the Deconvolutional Generative Model (DGM), which introduces a new regularizer (RPN) and a new loss function (Max-Min cross entropy). The result is that this approach exceeds or matches state-of-the-art performance on benchmarks like SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.
Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.