Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation
This work addresses a specific issue in NAS for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of ranking function variation in zero-shot Neural Architecture Search (NAS) by proposing a stochastic ordering approach that treats ranking outputs as random variables, which effectively boosts search performance on standard benchmarks.
Neural Architecture Search (NAS) is a powerful automatic alternative to manual design of a neural network. In the zero-shot version, a fast ranking function is used to compare architectures without training them. The outputs of the ranking functions often vary significantly due to different sources of randomness, including the evaluated architecture's weights' initialization or the batch of data used for calculations. A common approach to addressing the variation is to average a ranking function output over several evaluations. We propose taking into account the variation in a different manner, by viewing the ranking function output as a random variable representing a proxy performance metric. During the search process, we strive to construct a stochastic ordering of the performance metrics to determine the best architecture. Our experiments show that the proposed stochastic ordering can effectively boost performance of a search on standard benchmark search spaces.