Building a Regular Decision Boundary with Deep Networks
This work provides insights into neural network behavior for researchers, though it is incremental as it builds on existing architectures and focuses on empirical analysis rather than a new paradigm.
The authors tackled the problem of understanding empirical properties of neural networks by building a generic convolutional architecture, achieving up to 95.4% and 79.6% accuracy on CIFAR10 and CIFAR100, and showed that increasing network width can compete with deep networks while analyzing representation regularity.
In this work, we build a generic architecture of Convolutional Neural Networks to discover empirical properties of neural networks. Our first contribution is to introduce a state-of-the-art framework that depends upon few hyper parameters and to study the network when we vary them. It has no max pooling, no biases, only 13 layers, is purely convolutional and yields up to 95.4% and 79.6% accuracy respectively on CIFAR10 and CIFAR100. We show that the nonlinearity of a deep network does not need to be continuous, non expansive or point-wise, to achieve good performance. We show that increasing the width of our network permits being competitive with very deep networks. Our second contribution is an analysis of the contraction and separation properties of this network. Indeed, a 1-nearest neighbor classifier applied on deep features progressively improves with depth, which indicates that the representation is progressively more regular. Besides, we defined and analyzed local support vectors that separate classes locally. All our experiments are reproducible and code is available online, based on TensorFlow.