Zero-Shot NAS via the Suppression of Local Entropy Decrease
This work addresses the time-consuming architecture evaluation problem in NAS for researchers and practitioners, offering a significant speed-up while maintaining or improving accuracy, though it is incremental as it builds on existing zero-shot NAS methods.
The paper tackles the computational bottleneck in neural architecture search (NAS) by introducing a data-free and running-free proxy called suppression of local entropy decrease (SED), which uses architectural topologies to evaluate performance without backpropagation or input data, achieving state-of-the-art results on five benchmarks with computation time reduced by three orders of magnitude and selecting higher-accuracy architectures in one second.
Architecture performance evaluation is the most time-consuming part of neural architecture search (NAS). Zero-Shot NAS accelerates the evaluation by utilizing zero-cost proxies instead of training. Though effective, existing zero-cost proxies require invoking backpropagations or running networks on input data, making it difficult to further accelerate the computation of proxies. To alleviate this issue, architecture topologies are used to evaluate the performance of networks in this study. We prove that particular architectural topologies decrease the local entropy of feature maps, which degrades specific features to a bias, thereby reducing network performance. Based on this proof, architectural topologies are utilized to quantify the suppression of local entropy decrease (SED) as a data-free and running-free proxy. Experimental results show that SED outperforms most state-of-the-art proxies in terms of architecture selection on five benchmarks, with computation time reduced by three orders of magnitude. We further compare the SED-based NAS with state-of-the-art proxies. SED-based NAS selects the architecture with higher accuracy and fewer parameters in only one second. The theoretical analyses of local entropy and experimental results demonstrate that the suppression of local entropy decrease facilitates selecting optimal architectures in Zero-Shot NAS.