Neural Inheritance Relation Guided One-Shot Layer Assignment Search
This work addresses a specific bottleneck in neural architecture search for researchers, offering an incremental improvement by leveraging discovered inheritance relations to enhance efficiency.
The paper tackles the problem of layer assignment in neural architecture search by discovering a neural inheritance relation, where optimal assignments for deeper networks inherit from shallow ones, and proposes an efficient one-shot search method that reduces search space and achieves results consistent with optimal selections on CIFAR-100, with superior performance on Tiny-ImageNet and ImageNet under fixed computational budgets.
Layer assignment is seldom picked out as an independent research topic in neural architecture search. In this paper, for the first time, we systematically investigate the impact of different layer assignments to the network performance by building an architecture dataset of layer assignment on CIFAR-100. Through analyzing this dataset, we discover a neural inheritance relation among the networks with different layer assignments, that is, the optimal layer assignments for deeper networks always inherit from those for shallow networks. Inspired by this neural inheritance relation, we propose an efficient one-shot layer assignment search approach via inherited sampling. Specifically, the optimal layer assignment searched in the shallow network can be provided as a strong sampling priori to train and search the deeper ones in supernet, which extremely reduces the network search space. Comprehensive experiments carried out on CIFAR-100 illustrate the efficiency of our proposed method. Our search results are strongly consistent with the optimal ones directly selected from the architecture dataset. To further confirm the generalization of our proposed method, we also conduct experiments on Tiny-ImageNet and ImageNet. Our searched results are remarkably superior to the handcrafted ones under the unchanged computational budgets. The neural inheritance relation discovered in this paper can provide insights to the universal neural architecture search.