RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
This addresses the need for robust neural architectures against adversarial attacks, but it is incremental as it builds on existing NAS and knowledge distillation methods.
The paper tackles the problem of deep neural networks being vulnerable to adversarial attacks by proposing RNAS-CL, a neural architecture search algorithm that uses cross-layer knowledge distillation from a robust teacher to improve robustness, resulting in small and robust architectures as evidenced by experimental results.
Deep Neural Networks are vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the driving tools of deep neural networks, demonstrates superior performance in prediction accuracy in various machine learning applications. However, it is unclear how it performs against adversarial attacks. Given the presence of a robust teacher, it would be interesting to investigate if NAS would produce robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental result evidences the effectiveness of RNAS-CL and shows that RNAS-CL produces small and robust neural architecture.