Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
This addresses the need for efficient neural network deployment in dynamic resource environments, representing an incremental improvement over existing supernet training techniques.
The paper tackles the problem of training resource-adaptive supernets that adjust subnets for inference based on available resources, proposing prioritized subnet sampling (PSS-Net) to outperform state-of-the-art methods on ImageNet with MobileNet-V1/V2 and ResNet-50.
A resource-adaptive supernet adjusts its subnets for inference to fit the dynamically available resources. In this paper, we propose prioritized subnet sampling to train a resource-adaptive supernet, termed PSS-Net. We maintain multiple subnet pools, each of which stores the information of substantial subnets with similar resource consumption. Considering a resource constraint, subnets conditioned on this resource constraint are sampled from a pre-defined subnet structure space and high-quality ones will be inserted into the corresponding subnet pool. Then, the sampling will gradually be prone to sampling subnets from the subnet pools. Moreover, the one with a better performance metric is assigned with higher priority to train our PSS-Net, if sampling is from a subnet pool. At the end of training, our PSS-Net retains the best subnet in each pool to entitle a fast switch of high-quality subnets for inference when the available resources vary. Experiments on ImageNet using MobileNet-V1/V2 and ResNet-50 show that our PSS-Net can well outperform state-of-the-art resource-adaptive supernets. Our project is publicly available at https://github.com/chenbong/PSS-Net.