Critical initialisation in continuous approximations of binary neural networks
This work addresses the challenge of training deep binary neural networks, which is crucial for efficient AI deployment on resource-constrained devices, though it is incremental as it builds on existing surrogate methods.
The authors tackled the problem of training binary neural networks with ±1 weights and activations using continuous surrogate networks, deriving new surrogates via a Markov chain approach and analyzing their critical initializations to enable training of deeper networks, showing that initializing means close to ±1 improves performance.
The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to $\pm 1$, for deeper networks to be trainable.