Neural Architecture Generator Optimization
This work addresses the bottleneck in NAS for machine learning researchers by enabling more novel architectural discoveries through a more efficient search process, though it is incremental in improving existing NAS methods.
The paper tackles the problem of over-reliance on expert knowledge in Neural Architecture Search (NAS) by proposing a new hierarchical and graph-based search space that reduces dimensionality and expands architecture variety, resulting in lightweight yet highly competitive models on six benchmark datasets.
Neural Architecture Search (NAS) was first proposed to achieve state-of-the-art performance through the discovery of new architecture patterns, without human intervention. An over-reliance on expert knowledge in the search space design has however led to increased performance (local optima) without significant architectural breakthroughs, thus preventing truly novel solutions from being reached. In this work we 1) are the first to investigate casting NAS as a problem of finding the optimal network generator and 2) we propose a new, hierarchical and graph-based search space capable of representing an extremely large variety of network types, yet only requiring few continuous hyper-parameters. This greatly reduces the dimensionality of the problem, enabling the effective use of Bayesian Optimisation as a search strategy. At the same time, we expand the range of valid architectures, motivating a multi-objective learning approach. We demonstrate the effectiveness of this strategy on six benchmark datasets and show that our search space generates extremely lightweight yet highly competitive models.