Predict NAS Multi-Task by Stacking Ensemble Models using GP-NAS
This work addresses a specific challenge in NAS for computer vision researchers, though it appears incremental as it builds on existing ensemble and NAS methods.
The paper tackled the problem of predicting neural architecture search (NAS) performance with limited training data and multi-task correlations, achieving first place in the CVPR 2022 Track 2 Challenge using a stacking ensemble model with GP-NAS.
Accurately predicting the performance of architecture with small sample training is an important but not easy task. How to analysis and train dataset to overcome overfitting is the core problem we should deal with. Meanwhile if there is the mult-task problem, we should also think about if we can take advantage of their correlation and estimate as fast as we can. In this track, Super Network builds a search space based on ViT-Base. The search space contain depth, num-heads, mpl-ratio and embed-dim. What we done firstly are pre-processing the data based on our understanding of this problem which can reduce complexity of problem and probability of over fitting. Then we tried different kind of models and different way to combine them. Finally we choose stacking ensemble models using GP-NAS with cross validation. Our stacking model ranked 1st in CVPR 2022 Track 2 Challenge.