SplineNets: Continuous Neural Decision Graphs
This work addresses efficiency and accuracy challenges in deep learning for practitioners, though it appears incremental as it builds on existing neural decision graph concepts.
The paper tackles the problem of reducing runtime complexity and computation costs in convolutional neural networks (CNNs) by introducing SplineNets, a continuous generalization of neural decision graphs that conditions functions on input and computational path. The result shows that SplineNets can significantly increase accuracy with negligible speed cost, matching the precision of a 110-level ResNet with a 32-level SplineNet.
We present SplineNets, a practical and novel approach for using conditioning in convolutional neural networks (CNNs). SplineNets are continuous generalizations of neural decision graphs, and they can dramatically reduce runtime complexity and computation costs of CNNs, while maintaining or even increasing accuracy. Functions of SplineNets are both dynamic (i.e., conditioned on the input) and hierarchical (i.e., conditioned on the computational path). SplineNets employ a unified loss function with a desired level of smoothness over both the network and decision parameters, while allowing for sparse activation of a subset of nodes for individual samples. In particular, we embed infinitely many function weights (e.g. filters) on smooth, low dimensional manifolds parameterized by compact B-splines, which are indexed by a position parameter. Instead of sampling from a categorical distribution to pick a branch, samples choose a continuous position to pick a function weight. We further show that by maximizing the mutual information between spline positions and class labels, the network can be optimally utilized and specialized for classification tasks. Experiments show that our approach can significantly increase the accuracy of ResNets with negligible cost in speed, matching the precision of a 110 level ResNet with a 32 level SplineNet.