Learning Morphisms with Gauss-Newton Approximation for Growing Networks
This addresses the problem of computational efficiency in NAS for researchers and practitioners, but it is incremental as it builds on existing network morphism methods.
The paper tackled the challenge of efficiently determining where to grow networks in Neural Architecture Search (NAS) by proposing a method that uses a Gauss-Newton approximation to learn and evaluate candidate network morphisms, resulting in similar or better architectures at a smaller computational cost on CIFAR-10 and CIFAR-100 tasks.
A popular method for Neural Architecture Search (NAS) is based on growing networks via small local changes to the network's architecture called network morphisms. These methods start with a small seed network and progressively grow the network by adding new neurons in an automated way. However, it remains a challenge to efficiently determine which parts of the network are best to grow. Here we propose a NAS method for growing a network by using a Gauss-Newton approximation of the loss function to efficiently learn and evaluate candidate network morphisms. We compare our method with state of the art NAS methods for CIFAR-10 and CIFAR-100 classification tasks, and conclude our method learns similar quality or better architectures at a smaller computational cost.