DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
This work addresses the challenge of improving the reliability and performance of differentiable NAS methods for researchers and practitioners in automated machine learning, though it is incremental as it builds upon existing DARTS variants.
The paper tackles the problem of constrained bilevel optimization in Differentiable Architecture Search (DARTS) by introducing DARTS-PRIME, which uses dynamic scheduling and proximity regularization to improve architectural weight updates and discretization, resulting in enhanced performance and reliability comparable to state-of-the-art methods in multiple domains.
Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing and improving parts of the DARTS framework have recently been proposed. By considering the problem as a constrained bilevel optimization, we present and analyze DARTS-PRIME, a variant including improvements to architectural weight update scheduling and regularization towards discretization. We propose a dynamic schedule based on per-minibatch network information to make architecture updates more informed, as well as proximity regularization to promote well-separated discretization. Our results in multiple domains show that DARTS-PRIME improves both performance and reliability, comparable to state-of-the-art in differentiable NAS.