Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation
This work addresses computational inefficiency in CNNs for computer vision applications, representing an incremental improvement by optimizing existing methods rather than introducing a new paradigm.
The paper tackled spatial redundancy in CNN feature maps by computing features only at sparsely sampled locations based on activation responses and reconstructing the map via interpolation, saving substantial computation while maintaining accuracy across various computer vision tasks.
In the feature maps of CNNs, there commonly exists considerable spatial redundancy that leads to much repetitive processing. Towards reducing this superfluous computation, we propose to compute features only at sparsely sampled locations, which are probabilistically chosen according to activation responses, and then densely reconstruct the feature map with an efficient interpolation procedure. With this sampling-interpolation scheme, our network avoids expending computation on spatial locations that can be effectively interpolated, while being robust to activation prediction errors through broadly distributed sampling. A technical challenge of this sampling-based approach is that the binary decision variables for representing discrete sampling locations are non-differentiable, making them incompatible with backpropagation. To circumvent this issue, we make use of a reparameterization trick based on the Gumbel-Softmax distribution, with which backpropagation can iterate these variables towards binary values. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.