Incomplete Dot Products for Dynamic Computation Scaling in Neural Network Inference
This provides a convenient method for devices to dynamically lower computation costs based on system budgets, though it is incremental as it builds on existing channel selection techniques.
The paper tackles the problem of dynamic computation scaling in neural network inference by proposing incomplete dot products (IDP) to adjust the number of input channels per layer, allowing a single network to trade accuracy for reduced power and latency. For example, on CIFAR-10, IDP reduces computation by 75% without significant accuracy loss, and VGG-16 with 50% IDP achieves 70% accuracy compared to 35% with a standard reduced channel set.
We propose the use of incomplete dot products (IDP) to dynamically adjust the number of input channels used in each layer of a convolutional neural network during feedforward inference. IDP adds monotonically non-increasing coefficients, referred to as a "profile", to the channels during training. The profile orders the contribution of each channel in non-increasing order. At inference time, the number of channels used can be dynamically adjusted to trade off accuracy for lowered power consumption and reduced latency by selecting only a beginning subset of channels. This approach allows for a single network to dynamically scale over a computation range, as opposed to training and deploying multiple networks to support different levels of computation scaling. Additionally, we extend the notion to multiple profiles, each optimized for some specific range of computation scaling. We present experiments on the computation and accuracy trade-offs of IDP for popular image classification models and datasets. We demonstrate that, for MNIST and CIFAR-10, IDP reduces computation significantly, e.g., by 75%, without significantly compromising accuracy. We argue that IDP provides a convenient and effective means for devices to lower computation costs dynamically to reflect the current computation budget of the system. For example, VGG-16 with 50% IDP (using only the first 50% of channels) achieves 70% in accuracy on the CIFAR-10 dataset compared to the standard network which achieves only 35% accuracy when using the reduced channel set.