How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor
This work addresses the problem of making NAS more accessible and efficient for under-represented domains, though it is incremental as it builds on existing meta-learning methods applied to NAS.
The paper tackled the high cost and uncertainty of applying neural architecture search (NAS) to emerging domains by extracting transferable knowledge from public NAS benchmarks using meta-learning for few-shot adaptation, achieving superior or matching performance in cross-validation and successful extrapolation to new search spaces and tasks across 6 benchmarks and 16 settings.
Neural architecture search has proven to be a powerful approach to designing and refining neural networks, often boosting their performance and efficiency over manually-designed variations, but comes with computational overhead. While there has been a considerable amount of research focused on lowering the cost of NAS for mainstream tasks, such as image classification, a lot of those improvements stem from the fact that those tasks are well-studied in the broader context. Consequently, applicability of NAS to emerging and under-represented domains is still associated with a relatively high cost and/or uncertainty about the achievable gains. To address this issue, we turn our focus towards the recent growth of publicly available NAS benchmarks in an attempt to extract general NAS knowledge, transferable across different tasks and search spaces. We borrow from the rich field of meta-learning for few-shot adaptation and carefully study applicability of those methods to NAS, with a special focus on the relationship between task-level correlation (domain shift) and predictor transferability; which we deem critical for improving NAS on diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction, spanning in total 16 NAS settings -- our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.