Channel selection using Gumbel Softmax
This addresses efficiency needs for applications like mobile computing, though it is incremental as it builds on existing pruning and conditional computation methods.
The paper tackles the problem of reducing computational costs for neural network inference by proposing an end-to-end framework that optimizes the tradeoff between accuracy and speed, achieving 45-52% less computation on ImageNet classification with ResNet.
Important applications such as mobile computing require reducing the computational costs of neural network inference. Ideally, applications would specify their preferred tradeoff between accuracy and speed, and the network would optimize this end-to-end, using classification error to remove parts of the network. Increasing speed can be done either during training - e.g., pruning filters - or during inference - e.g., conditionally executing a subset of the layers. We propose a single end-to-end framework that can improve inference efficiency in both settings. We use a combination of batch activation loss and classification loss, and Gumbel reparameterization to learn network structure. We train end-to-end, and the same technique supports pruning as well as conditional computation. We obtain promising experimental results for ImageNet classification with ResNet (45-52% less computation).