The High-Dimensional Geometry of Binary Neural Networks
This provides a foundational explanation for BNNs, which are important for compressing neural networks to improve efficiency in resource-constrained applications, though it is incremental as it builds on prior empirical work.
The paper tackles the lack of theoretical understanding for why binary neural networks (BNNs) with binary weights and activations can effectively capture data features, showing that their success is due to the high-dimensional geometry of binary vectors preserving dot products with ideal continuous vectors.
Recent research has shown that one can train a neural network with binary weights and activations at train time by augmenting the weights with a high-precision continuous latent variable that accumulates small changes from stochastic gradient descent. However, there is a dearth of theoretical analysis to explain why we can effectively capture the features in our data with binary weights and activations. Our main result is that the neural networks with binary weights and activations trained using the method of Courbariaux, Hubara et al. (2016) work because of the high-dimensional geometry of binary vectors. In particular, the ideal continuous vectors that extract out features in the intermediate representations of these BNNs are well-approximated by binary vectors in the sense that dot products are approximately preserved. Compared to previous research that demonstrated the viability of such BNNs, our work explains why these BNNs work in terms of the HD geometry. Our theory serves as a foundation for understanding not only BNNs but a variety of methods that seek to compress traditional neural networks. Furthermore, a better understanding of multilayer binary neural networks serves as a starting point for generalizing BNNs to other neural network architectures such as recurrent neural networks.