Pareto-aware Neural Architecture Generation for Diverse Computational Budgets
This addresses the problem of deploying deep models efficiently under diverse computational constraints, offering a more scalable solution for real-world applications.
The paper tackles the inefficiency of independent neural architecture searches for different computational budgets by proposing a Pareto-aware Neural Architecture Generator (PNAG) that trains once and dynamically generates optimal architectures for any budget, achieving superior results across three hardware platforms.
Designing feasible and effective architectures under diverse computational budgets, incurred by different applications/devices, is essential for deploying deep models in real-world applications. To achieve this goal, existing methods often perform an independent architecture search process for each target budget, which is very inefficient yet unnecessary. More critically, these independent search processes cannot share their learned knowledge (i.e., the distribution of good architectures) with each other and thus often result in limited search results. To address these issues, we propose a Pareto-aware Neural Architecture Generator (PNAG) which only needs to be trained once and dynamically produces the Pareto optimal architecture for any given budget via inference. To train our PNAG, we learn the whole Pareto frontier by jointly finding multiple Pareto optimal architectures under diverse budgets. Such a joint search algorithm not only greatly reduces the overall search cost but also improves the search results. Extensive experiments on three hardware platforms (i.e., mobile device, CPU, and GPU) show the superiority of our method over existing methods.