Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Between
This work addresses the need for a standardized framework in NAS research, enabling systematic exploration and fair comparisons across diverse neural architectures.
The authors tackled the problem of fragmented neural architecture search (NAS) by introducing UniNAS, a unified search space that covers convolutional networks, transformers, and hybrids, and demonstrated that discovered architectures outperform state-of-the-art hand-crafted ones under identical training setups.
We introduce Universal Neural Architecture Space (UniNAS), a generic search space for neural architecture search (NAS) which unifies convolutional networks, transformers, and their hybrid architectures under a single, flexible framework. Our approach enables discovery of novel architectures as well as analyzing existing architectures in a common framework. We also propose a new search algorithm that allows traversing the proposed search space, and demonstrate that the space contains interesting architectures, which, when using identical training setup, outperform state-of-the-art hand-crafted architectures. Finally, a unified toolkit including a standardized training and evaluation protocol is introduced to foster reproducibility and enable fair comparison in NAS research. Overall, this work opens a pathway towards systematically exploring the full spectrum of neural architectures with a unified graph-based NAS perspective.