Neural Architecture Search: Insights from 1000 Papers
It provides a structured overview for researchers and practitioners in machine learning, but it is incremental as a survey rather than presenting new methods.
This survey paper organizes and comprehensively guides neural architecture search (NAS), which automates the design of neural architectures and has outpaced human-designed ones on many tasks, based on insights from over 1000 papers released since 2020.
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.