Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS
This work addresses the initialization bottleneck in NAS for researchers and practitioners, offering an incremental improvement over existing stochastic methods.
The paper tackles the problem of inefficient initialization in neural architecture search (NAS) by proposing a data-driven method using clustering analysis and centroid extraction to initialize NAS algorithms, resulting in faster convergence and better final performance compared to random initialization on NAS-bench-101.
Lots of effort in neural architecture search (NAS) research has been dedicated to algorithmic development, aiming at designing more efficient and less costly methods. Nonetheless, the investigation of the initialization of these techniques remain scare, and currently most NAS methodologies rely on stochastic initialization procedures, because acquiring information prior to search is costly. However, the recent availability of NAS benchmarks have enabled low computational resources prototyping. In this study, we propose to accelerate a NAS algorithm using a data-driven initialization technique, leveraging the availability of NAS benchmarks. Particularly, we proposed a two-step methodology. First, a calibrated clustering analysis of the search space is performed. Second, the centroids are extracted and used to initialize a NAS algorithm. We tested our proposal using Aging Evolution, an evolutionary algorithm, on NAS-bench-101. The results show that, compared to a random initialization, a faster convergence and a better performance of the final solution is achieved.