Evolutionary Neural Architecture Search with Dual Contrastive Learning
This work addresses the computational bottleneck in ENAS for researchers and practitioners in automated machine learning, offering an incremental improvement over existing methods.
This paper tackles the high computational cost of training neural predictors in Evolutionary Neural Architecture Search (ENAS) by introducing DCL-ENAS, which uses dual contrastive learning to train the predictor without requiring many fully trained architectures. It achieves state-of-the-art validation accuracy improvements of 0.05% to 0.39% on benchmarks and a 2.5 percentage point gain on a real-world ECG task with only 7.7 GPU-days.
Evolutionary Neural Architecture Search (ENAS) has gained attention for automatically designing neural network architectures. Recent studies use a neural predictor to guide the process, but the high computational costs of gathering training data -- since each label requires fully training an architecture -- make achieving a high-precision predictor with { limited compute budget (i.e., a capped number of fully trained architecture-label pairs)} crucial for ENAS success. This paper introduces ENAS with Dual Contrastive Learning (DCL-ENAS), a novel method that employs two stages of contrastive learning to train the neural predictor. In the first stage, contrastive self-supervised learning is used to learn meaningful representations from neural architectures without requiring labels. In the second stage, fine-tuning with contrastive learning is performed to accurately predict the relative performance of different architectures rather than their absolute performance, which is sufficient to guide the evolutionary search. Across NASBench-101 and NASBench-201, DCL-ENAS achieves the highest validation accuracy, surpassing the strongest published baselines by 0.05\% (ImageNet16-120) to 0.39\% (NASBench-101). On a real-world ECG arrhythmia classification task, DCL-ENAS improves performance by approximately 2.5 percentage points over a manually designed, non-NAS model obtained via random search, while requiring only 7.7 GPU-days.