MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks
This addresses the challenge of efficient neural network design for practitioners needing optimized models under computational constraints, representing an incremental improvement over prior methods.
The authors tackled the problem of automating neural network structure design under resource constraints, resulting in a method that discovers novel architectures achieving higher performance while respecting specified limits like FLOPs.
We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.