Tackling Neural Architecture Search With Quality Diversity Optimization
This addresses the need for more effective NAS methods that find diverse, high-performing architectures tailored to specific hardware constraints, representing an incremental improvement in the field.
The authors tackled the discrepancy between practical multi-objective neural architecture search (NAS) and its optimization formulation by reframing it as a quality diversity optimization (QDO) problem, introducing three optimizers that outperform multi-objective NAS in solution quality and efficiency.
Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact. While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error. Although considerable progress has been made in the field of multi-objective NAS, we argue that there is some discrepancy between the actual optimization problem of practical interest and the optimization problem that multi-objective NAS tries to solve. We resolve this discrepancy by formulating the multi-objective NAS problem as a quality diversity optimization (QDO) problem and introduce three quality diversity NAS optimizers (two of them belonging to the group of multifidelity optimizers), which search for high-performing yet diverse architectures that are optimal for application-specific niches, e.g., hardware constraints. By comparing these optimizers to their multi-objective counterparts, we demonstrate that quality diversity NAS in general outperforms multi-objective NAS with respect to quality of solutions and efficiency. We further show how applications and future NAS research can thrive on QDO.