Median Binary-Connect Method and a Binary Convolutional Neural Nework for Word Recognition
This work addresses the challenge of efficient neural network deployment on resource-constrained devices like mobile phones, offering an incremental improvement in binary network training for specific applications.
The authors tackled the problem of training binary weight convolutional neural networks by proposing a new projection formula using the l_1 norm with a median computation, which improved performance over the standard l_2 norm method. Experiments on a 10-keyword classification task showed that this median BinaryConnect method achieved 92.4% test accuracy, only 1.1% lower than the full-precision network, and doubled the speed on an Android phone for spoken keyword recognition.
We propose and study a new projection formula for training binary weight convolutional neural networks. The projection formula measures the error in approximating a full precision (32 bit) vector by a 1-bit vector in the l_1 norm instead of the standard l_2 norm. The l_1 projector is in closed analytical form and involves a median computation instead of an arithmatic average in the l_2 projector. Experiments on 10 keywords classification show that the l_1 (median) BinaryConnect (BC) method outperforms the regular BC, regardless of cold or warm start. The binary network trained by median BC and a recent blending technique reaches test accuracy 92.4%, which is 1.1% lower than the full-precision network accuracy 93.5%. On Android phone app, the trained binary network doubles the speed of full-precision network in spoken keywords recognition.