Speeding Up Neural Networks for Large Scale Classification using WTA Hashing
This addresses the problem of high computational cost for researchers and practitioners training neural networks on datasets with many output classes, though it appears incremental as it builds on existing hashing techniques.
The authors tackled the computational bottleneck of training convolutional neural networks on large-scale classification tasks by applying Winner Takes All hashing to fully connected layers, achieving a 7-fold speedup during training with no performance drop.
In this paper we propose to use the Winner Takes All hashing technique to speed up forward propagation and backward propagation in fully connected layers in convolutional neural networks. The proposed technique reduces significantly the computational complexity, which in turn, allows us to train layers with a large number of kernels with out the associated time penalty. As a consequence we are able to train convolutional neural network on a very large number of output classes with only a small increase in the computational cost. To show the effectiveness of the technique we train a new output layer on a pretrained network using both the regular multiplicative approach and our proposed hashing methodology. Our results showed no drop in performance and demonstrate, with our implementation, a 7 fold speed up during the training.