FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
This work addresses the need for resource-efficient inference tailored to specific system constraints, such as GPUs vs. mobile devices, though it is incremental as it extends an existing state-of-the-art technique.
The authors tackled the problem of training sparse neural networks by directly incorporating FLOPs as an optimization objective, enabling practitioners to specify target FLOPs for model compression, and demonstrated successful training of different networks for image classification under given FLOPs requirements.
There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporting FLOPs as part of the results. Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification.