A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search
It provides a comprehensive review for researchers in NAS, but is incremental as it synthesizes existing work without new results.
This survey categorizes and evaluates supernet optimization methods for Neural Architecture Search, analyzing spatial and temporal approaches to improve architecture selection efficiency.
This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.