SMASH: One-Shot Model Architecture Search through HyperNetworks
This addresses the need for faster architecture design in deep learning, though it is incremental as it builds on existing hypernetwork and architecture search concepts.
The paper tackles the problem of computationally expensive neural architecture search by proposing SMASH, a method that uses a HyperNet to generate weights for different architectures, enabling efficient search with a single training run and achieving competitive performance on datasets like CIFAR-10 and ImageNet32x32.
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASH