DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation
This addresses the problem of inefficient and inflexible neural architecture search for machine learning practitioners, offering a more adaptable and cost-effective solution, though it is incremental as it builds on existing NAS methods.
The paper tackles the problem of neural architecture search (NAS) being resource-intensive and lacking adaptability across different scenarios and hardware, proposing DANCE which reformulates NAS as a continuous evolution problem. The result shows that DANCE consistently outperforms state-of-the-art NAS methods in accuracy while significantly reducing search costs, with robust performance under varying computational constraints.
Neural Architecture Search (NAS) has emerged as a powerful approach for automating neural network design. However, existing NAS methods face critical limitations in real-world deployments: architectures lack adaptability across scenarios, each deployment context requires costly separate searches, and performance consistency across diverse platforms remains challenging. We propose DANCE (Dynamic Architectures with Neural Continuous Evolution), which reformulates architecture search as a continuous evolution problem through learning distributions over architectural components. DANCE introduces three key innovations: a continuous architecture distribution enabling smooth adaptation, a unified architecture space with learned selection gates for efficient sampling, and a multi-stage training strategy for effective deployment optimization. Extensive experiments across five datasets demonstrate DANCE's effectiveness. Our method consistently outperforms state-of-the-art NAS approaches in terms of accuracy while significantly reducing search costs. Under varying computational constraints, DANCE maintains robust performance while smoothly adapting architectures to different hardware requirements. The code and appendix can be found at https://github.com/Applied-Machine-Learning-Lab/DANCE.