AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
This work addresses the need for faster and more efficient NAS to deploy DNNs on mobile and edge devices, offering a novel approach that improves upon existing methods but is incremental in the context of NAS advancements.
The paper tackles the problem of inefficient neural architecture search (NAS) for resource-constrained devices by proposing AutoShrink, a topology-aware method that uses edge shrinking in DAGs to optimize cell structures, resulting in up to 48% parameter reduction and 34% MACs savings on ImageNet-1K with comparable accuracy, while reducing search time by 7.2x for CNNs and 6.7x for RNNs.
Resource is an important constraint when deploying Deep Neural Networks (DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based search approach, which limits the flexibility of network patterns in learned cell structures. Moreover, due to the topology-agnostic nature of existing works, including both cell-based and node-based approaches, the search process is time consuming and the performance of found architecture may be sub-optimal. To address these problems, we propose AutoShrink, a topology-aware Neural Architecture Search(NAS) for searching efficient building blocks of neural architectures. Our method is node-based and thus can learn flexible network patterns in cell structures within a topological search space. Directed Acyclic Graphs (DAGs) are used to abstract DNN architectures and progressively optimize the cell structure through edge shrinking. As the search space intrinsically reduces as the edges are progressively shrunk, AutoShrink explores more flexible search space with even less search time. We evaluate AutoShrink on image classification and language tasks by crafting ShrinkCNN and ShrinkRNN models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34% Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are crafted within 1.5 GPU hours, which is 7.2x and 6.7x faster than the crafting time of SOTA CNN and RNN models, respectively.