Divide-and-Conquer the NAS puzzle in Resource Constrained Federated Learning Systems
This addresses the challenge of efficient neural architecture search in resource-constrained federated learning systems, which is an incremental improvement for edge device applications.
The paper tackles the problem of designing custom neural architectures in federated learning systems by proposing DC-NAS, a divide-and-conquer approach that uses a novel diversified sampling strategy and channel pruning to reduce resource usage. It achieves near iso-accuracy compared to full-scale federated NAS with 50% fewer resources on benchmarks like CIFAR10, CIFAR100, EMNIST, and TinyImagenet.
Federated Learning (FL) is a privacy-preserving distributed machine learning approach geared towards applications in edge devices. However, the problem of designing custom neural architectures in federated environments is not tackled from the perspective of overall system efficiency. In this paper, we propose DC-NAS -- a divide-and-conquer approach that performs supernet-based Neural Architecture Search (NAS) in a federated system by systematically sampling the search space. We propose a novel diversified sampling strategy that balances exploration and exploitation of the search space by initially maximizing the distance between the samples and progressively shrinking this distance as the training progresses. We then perform channel pruning to reduce the training complexity at the devices further. We show that our approach outperforms several sampling strategies including Hadamard sampling, where the samples are maximally separated. We evaluate our method on the CIFAR10, CIFAR100, EMNIST, and TinyImagenet benchmarks and show a comprehensive analysis of different aspects of federated learning such as scalability, and non-IID data. DC-NAS achieves near iso-accuracy as compared to full-scale federated NAS with 50% fewer resources.