Across-Task Neural Architecture Search via Meta Learning
This work addresses the problem of efficient neural architecture search for meta-learning in resource-constrained settings, offering a novel hybrid method that is incremental in combining existing techniques.
The paper tackles the challenge of applying neural architecture search (NAS) in meta-learning scenarios with limited compute and data by proposing AT-NAS, which combines gradient-based meta-learning with evolutionary algorithm-based NAS to learn over task distributions. Empirical results show that AT-NAS surpasses related approaches in few-shot classification accuracy and achieves performance comparable to models searched from scratch by adapting architectures in less than an hour from a pretrained meta-network that originally took 5 GPU-days.
Adequate labeled data and expensive compute resources are the prerequisites for the success of neural architecture search(NAS). It is challenging to apply NAS in meta-learning scenarios with limited compute resources and data. In this paper, an across-task neural architecture search (AT-NAS) is proposed to address the problem through combining gradient-based meta-learning with EA-based NAS to learn over the distribution of tasks. The supernet is learned over an entire set of tasks by meta-learning its weights. Architecture encodes of subnets sampled from the supernet are iteratively adapted by evolutionary algorithms while simultaneously searching for a task-sensitive meta-network. Searched meta-network can be adapted to a novel task via a few learning steps and only costs a little search time. Empirical results show that AT-NAS surpasses the related approaches on few-shot classification accuracy. The performance of AT-NAS on classification benchmarks is comparable to that of models searched from scratch, by adapting the architecture in less than an hour from a 5-GPU-day pretrained meta-network.