Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting
This addresses the challenge of maintaining quality of service for edge and IoT devices with variable computation and memory resources, offering an incremental improvement in sparse neural network design.
The paper tackles the problem of deploying deep neural networks on edge and IoT devices with fluctuating resources by proposing a sparse model that supports multiple sparsity levels, enabling dynamic selection during inference. It achieves sparse sub-models with an average of 13.38% weights and 14.97% FLOPs while maintaining accuracy comparable to dense models, and more-sparse subsets with 5.38% weights and 4.47% FLOPs incur only a 3.25% relative accuracy loss.
Deep neural networks (DNNs) have shown to provide superb performance in many real life applications, but their large computation cost and storage requirement have prevented them from being deployed to many edge and internet-of-things (IoT) devices. Sparse deep neural networks, whose majority weight parameters are zeros, can substantially reduce the computation complexity and memory consumption of the models. In real-use scenarios, devices may suffer from large fluctuations of the available computation and memory resources under different environment, and the quality of service (QoS) is difficult to maintain due to the long tail inferences with large latency. Facing the real-life challenges, we propose to train a sparse model that supports multiple sparse levels. That is, a hierarchical structure of weights are satisfied such that the locations and the values of the non-zero parameters of the more-sparse sub-model area subset of the less-sparse sub-model. In this way, one can dynamically select the appropriate sparsity level during inference, while the storage cost is capped by the least sparse sub-model. We have verified our methodologies on a variety of DNN models and tasks, including the ResNet-50, PointNet++, GNMT, and graph attention networks. We obtain sparse sub-models with an average of 13.38% weights and 14.97% FLOPs, while the accuracies are as good as their dense counterparts. More-sparse sub-models with 5.38% weights and 4.47% of FLOPs, which are subsets of the less-sparse ones, can be obtained with only 3.25% relative accuracy loss.