Revisiting Neural Architecture Search
This work addresses the challenge of reducing human effort in NAS for machine learning researchers and practitioners, representing a step towards more automated machine learning, though it is incremental in improving automation rather than a paradigm shift.
The paper tackles the problem of Neural Architecture Search (NAS) being heavily dependent on manual effort in search space design, proposing ReNAS, a novel approach that searches for complete neural networks automatically, achieving performance on-par with state-of-the-art methods that use handcrafted blocks.
Neural Architecture Search (NAS) is a collection of methods to craft the way neural networks are built. Current NAS methods are far from ab initio and automatic, as they use manual backbone architectures or micro building blocks (cells), which have had minor breakthroughs in performance compared to random baselines. They also involve a significant manual expert effort in various components of the NAS pipeline. This raises a natural question - Are the current NAS methods still heavily dependent on manual effort in the search space design and wiring like it was done when building models before the advent of NAS? In this paper, instead of merely chasing slight improvements over state-of-the-art (SOTA) performance, we revisit the fundamental approach to NAS and propose a novel approach called ReNAS that can search for the complete neural network without much human effort and is a step closer towards AutoML-nirvana. Our method starts from a complete graph mapped to a neural network and searches for the connections and operations by balancing the exploration and exploitation of the search space. The results are on-par with the SOTA performance with methods that leverage handcrafted blocks. We believe that this approach may lead to newer NAS strategies for a variety of network types.