AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
This addresses the heuristic and labor-intensive process of designing network depth for researchers and practitioners, though it is incremental as it builds on existing architectures and policies.
The paper tackles the problem of automating depth discovery in deep convolutional networks by proposing AutoGrow, which starts from a shallow seed architecture and grows layers based on accuracy improvements, stopping when no further gains are made. The result shows that AutoGrow discovers near-optimal depths across various datasets, such as achieving a better accuracy-computation trade-off in ResNets than human experts.
Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at https://github.com/wenwei202/autogrow.