Multi-objective Differentiable Neural Architecture Search
This work addresses the problem of computationally expensive Pareto front profiling in NAS for researchers and practitioners needing to balance performance and hardware metrics across devices, representing an incremental improvement over prior methods.
The paper tackles the challenge of efficiently profiling the Pareto front in multi-objective neural architecture search (NAS) by proposing a novel algorithm that uses a hypernetwork to encode user preferences, enabling a single search run to yield diverse architectures across multiple devices and objectives. The method outperforms existing MOO NAS methods across various search spaces and datasets, such as MobileNetV3 on ImageNet-1k, without additional costs.
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.