Neural Architecture Search via Bregman Iterations
This work addresses architecture optimization in deep learning, but it appears incremental as it builds on existing NAS methods with a novel iterative approach.
The paper tackles the problem of Neural Architecture Search (NAS) by introducing a gradient-based one-shot algorithm that uses Bregman iterations to gradually add parameters, enabling the network to select optimal architectures for tasks like denoising, deblurring, and classification.
We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Starting from a sparse neural network our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner. This allows the network to choose the best architecture in the search space which makes it well-designed for a given task, e.g., by adding neurons or skip connections. We demonstrate that using our approach one can unveil, for instance, residual autoencoders for denoising, deblurring, and classification tasks. Code is available at https://github.com/TimRoith/BregmanLearning.