AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks
This work addresses the challenge of efficient training for binary neural networks, which is important for deploying models on resource-constrained devices, but it appears incremental as it builds upon established straight-through estimator techniques.
The paper tackles the problem of training deep neural networks with binary weights by proposing AdaSTE, an adaptive straight-through estimator derived from a bilevel optimization formulation, which achieves favorable performance compared to existing methods.
We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in particular with the straight-through gradient estimator successfully employed in BinaryConnect and subsequent methods. In fact, our proposed method can be interpreted as an adaptive variant of the original straight-through estimator that conditionally (but not always) acts like a linear mapping in the backward pass of error propagation. Experimental results demonstrate that our new algorithm offers favorable performance compared to existing approaches.