CONetV2: Efficient Auto-Channel Size Optimization for CNNs
This work addresses the challenge of reducing computational costs in Neural Architecture Search for CNNs, which is incremental as it builds on existing NAS methods by focusing on micro-search spaces.
The paper tackles the problem of efficiently optimizing channel sizes in CNNs under computational constraints by introducing an automated algorithm that extracts layer dependencies, uses knowledge distillation, and a novel metric for layer-wise analysis, resulting in architectures that outperform baselines by a large margin.
Neural Architecture Search (NAS) has been pivotal in finding optimal network configurations for Convolution Neural Networks (CNNs). While many methods explore NAS from a global search-space perspective, the employed optimization schemes typically require heavy computational resources. This work introduces a method that is efficient in computationally constrained environments by examining the micro-search space of channel size. In tackling channel-size optimization, we design an automated algorithm to extract the dependencies within different connected layers of the network. In addition, we introduce the idea of knowledge distillation, which enables preservation of trained weights, admist trials where the channel sizes are changing. Further, since the standard performance indicators (accuracy, loss) fail to capture the performance of individual network components (providing an overall network evaluation), we introduce a novel metric that highly correlates with test accuracy and enables analysis of individual network layers. Combining dependency extraction, metrics, and knowledge distillation, we introduce an efficient searching algorithm, with simulated annealing inspired stochasticity, and demonstrate its effectiveness in finding optimal architectures that outperform baselines by a large margin.