Combining Neural Architecture Search and Automatic Code Optimization: A Survey
This is an incremental survey that addresses efficiency challenges for deploying deep learning models on hardware-limited systems.
This survey tackles the problem of accelerating deep learning models on resource-constrained devices by exploring the combination of Hardware-aware Neural Architecture Search (HW-NAS) and Automatic Code Optimization (ACO) into a joint framework called NACOS, highlighting their sub-optimality when used independently.
Deep Learning models have experienced exponential growth in complexity and resource demands in recent years. Accelerating these models for efficient execution on resource-constrained devices has become more crucial than ever. Two notable techniques employed to achieve this goal are Hardware-aware Neural Architecture Search (HW-NAS) and Automatic Code Optimization (ACO). HW-NAS automatically designs accurate yet hardware-friendly neural networks, while ACO involves searching for the best compiler optimizations to apply on neural networks for efficient mapping and inference on the target hardware. This survey explores recent works that combine these two techniques within a single framework. We present the fundamental principles of both domains and demonstrate their sub-optimality when performed independently. We then investigate their integration into a joint optimization process that we call Hardware Aware-Neural Architecture and Compiler Optimizations co-Search (NACOS).