Entropic Score metric: Decoupling Topology and Size in Training-free NAS
This addresses the problem of automating neural network design for resource-constrained edge applications, representing a novel method for a known bottleneck.
The paper tackled the high computational cost of Neural Architecture Search (NAS) by introducing a training-free metric, Entropic Score, and a cyclic search algorithm to separately optimize model topology and size, achieving the fastest and most accurate NAS method for ImageNet classification in under 1 GPU hour.
Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.