Designed Dithering Sign Activation for Binary Neural Networks
This work addresses a specific bottleneck in binary neural networks for computer vision tasks, offering an incremental improvement in activation design.
The paper tackles the loss of fine-grained details in binary neural networks due to abrupt binarization in common Sign activation functions by proposing a designed dithering Sign activation that applies multiple thresholds using a spatially periodic kernel, achieving effective classification without increasing computational cost.
Binary Neural Networks emerged as a cost-effective and energy-efficient solution for computer vision tasks by binarizing either network weights or activations. However, common binary activations, such as the Sign activation function, abruptly binarize the values with a single threshold, losing fine-grained details in the feature outputs. This work proposes an activation that applies multiple thresholds following dithering principles, shifting the Sign activation function for each pixel according to a spatially periodic threshold kernel. Unlike literature methods, the shifting is defined jointly for a set of adjacent pixels, taking advantage of spatial correlations. Experiments over the classification task demonstrate the effectiveness of the designed dithering Sign activation function as an alternative activation for binary neural networks, without increasing the computational cost. Further, DeSign balances the preservation of details with the efficiency of binary operations.