OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search
This work addresses the efficiency bottleneck in NAS for researchers and practitioners in domains like computer vision and NLP, though it appears incremental as it builds on existing proxy-based methods.
The authors tackled the computationally expensive and discrete search space problem in neural architecture search (NAS) by proposing OptiProxy-NAS, an end-to-end optimization framework that reformulates the space to be continuous and differentiable, achieving superior search results and efficiency across 12 NAS tasks in 4 search spaces.
Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.