Compacting Binary Neural Networks by Sparse Kernel Selection
This work addresses efficiency and performance issues in BNNs for resource-constrained applications, representing an incremental improvement over existing methods.
The paper tackles the problem of compacting Binary Neural Networks (BNNs) by leveraging the power-law distribution of binary kernels to select a smaller subset of codewords, resulting in reduced model size and bit-wise computational costs while achieving accuracy improvements compared to state-of-the-art BNNs under comparable budgets.
Binary Neural Network (BNN) represents convolution weights with 1-bit values, which enhances the efficiency of storage and computation. This paper is motivated by a previously revealed phenomenon that the binary kernels in successful BNNs are nearly power-law distributed: their values are mostly clustered into a small number of codewords. This phenomenon encourages us to compact typical BNNs and obtain further close performance through learning non-repetitive kernels within a binary kernel subspace. Specifically, we regard the binarization process as kernel grouping in terms of a binary codebook, and our task lies in learning to select a smaller subset of codewords from the full codebook. We then leverage the Gumbel-Sinkhorn technique to approximate the codeword selection process, and develop the Permutation Straight-Through Estimator (PSTE) that is able to not only optimize the selection process end-to-end but also maintain the non-repetitive occupancy of selected codewords. Experiments verify that our method reduces both the model size and bit-wise computational costs, and achieves accuracy improvements compared with state-of-the-art BNNs under comparable budgets.