XferNAS: Transfer Neural Architecture Search
This work addresses the problem of slow NAS optimization for researchers and practitioners, offering a generally applicable and efficient solution that is incremental but highly effective.
The authors tackled the high computational cost of Neural Architecture Search (NAS) by proposing a transfer learning framework that reuses knowledge from previous searches, reducing search time from 200 to 6 GPU days (a 33x speed-up) and achieving new state-of-the-art results on CIFAR benchmarks.
The term Neural Architecture Search (NAS) refers to the automatic optimization of network architectures for a new, previously unknown task. Since testing an architecture is computationally very expensive, many optimizers need days or even weeks to find suitable architectures. However, this search time can be significantly reduced if knowledge from previous searches on different tasks is reused. In this work, we propose a generally applicable framework that introduces only minor changes to existing optimizers to leverage this feature. As an example, we select an existing optimizer and demonstrate the complexity of the integration of the framework as well as its impact. In experiments on CIFAR-10 and CIFAR-100, we observe a reduction in the search time from 200 to only 6 GPU days, a speed up by a factor of 33. In addition, we observe new records of 1.99 and 14.06 for NAS optimizers on the CIFAR benchmarks, respectively. In a separate study, we analyze the impact of the amount of source and target data. Empirically, we demonstrate that the proposed framework generally gives better results and, in the worst case, is just as good as the unmodified optimizer.