Nonlocal optimization of binary neural networks
This work addresses the challenge of optimizing BNNs, which are important for efficient deep learning on resource-constrained devices, but it appears incremental as it builds on existing message-passing algorithms.
The paper tackled the problem of training Binary Neural Networks (BNNs) by framing it as a discrete variable inference problem and proposed stochastic versions of Belief Propagation and Survey Propagation to address intractability, resulting in better parameter configurations compared to traditional gradient methods.
We explore training Binary Neural Networks (BNNs) as a discrete variable inference problem over a factor graph. We study the behaviour of this conversion in an under-parameterized BNN setting and propose stochastic versions of Belief Propagation (BP) and Survey Propagation (SP) message passing algorithms to overcome the intractability of their current formulation. Compared to traditional gradient methods for BNNs, our results indicate that both stochastic BP and SP find better configurations of the parameters in the BNN.