Bit Error Tolerance Metrics for Binarized Neural Networks
This research aims to provide a foundational understanding of bit error tolerance in BNNs, which is crucial for developing more scalable and efficient approximate memory systems for neural network inference.
This paper investigates the internal changes caused by bit flip training in binarized neural networks (BNNs) to understand their bit error tolerance. The authors propose two metrics: a neuron-level metric measuring the margin between pre-activation values and batch normalization thresholds, and an inter-neuron metric capturing the variance of neuron importance. Experimental results indicate a strong correlation between these metrics and bit error tolerance.
To reduce the resource demand of neural network (NN) inference systems, it has been proposed to use approximate memory, in which the supply voltage and the timing parameters are tuned trading accuracy with energy consumption and performance. Tuning these parameters aggressively leads to bit errors, which can be tolerated by NNs when bit flips are injected during training. However, bit flip training, which is the state of the art for achieving bit error tolerance, does not scale well; it leads to massive overheads and cannot be applied for high bit error rates (BERs). Alternative methods to achieve bit error tolerance in NNs are needed, but the underlying principles behind the bit error tolerance of NNs have not been reported yet. With this lack of understanding, further progress in the research on NN bit error tolerance will be restrained. In this study, our objective is to investigate the internal changes in the NNs that bit flip training causes, with a focus on binarized NNs (BNNs). To this end, we quantify the properties of bit error tolerant BNNs with two metrics. First, we propose a neuron-level bit error tolerance metric, which calculates the margin between the pre-activation values and batch normalization thresholds. Secondly, to capture the effects of bit error tolerance on the interplay of neurons, we propose an inter-neuron bit error tolerance metric, which measures the importance of each neuron and computes the variance over all importance values. Our experimental results support that these two metrics are strongly related to bit error tolerance.