Dynamical Isometry based Rigorous Fair Neural Architecture Search
This addresses fairness and interpretability issues in neural architecture search for researchers and practitioners, though it is incremental as it builds on existing weight-sharing techniques.
The paper tackles the lack of interpretability and fairness in weight-sharing neural architecture search by proposing a method based on dynamical isometry, which achieves state-of-the-art top-1 validation accuracy on ImageNet classification with the same model size.
Recently, the weight-sharing technique has significantly speeded up the training and evaluation procedure of neural architecture search. However, most existing weight-sharing strategies are solely based on experience or observation, which makes the searching results lack interpretability and rationality. In addition, due to the negligence of fairness, current methods are prone to make misjudgments in module evaluation. To address these problems, we propose a novel neural architecture search algorithm based on dynamical isometry. We use the fix point analysis method in the mean field theory to analyze the dynamics behavior in the steady state random neural network, and how dynamic isometry guarantees the fairness of weight-sharing based NAS. Meanwhile, we prove that our module selection strategy is rigorous fair by estimating the generalization error of all modules with well-conditioned Jacobian. Extensive experiments show that, with the same size, the architecture searched by the proposed method can achieve state-of-the-art top-1 validation accuracy on ImageNet classification. In addition, we demonstrate that our method is able to achieve better and more stable training performance without loss of generality.