BiBench: Benchmarking and Analyzing Network Binarization
This work addresses the need for a comprehensive benchmark to analyze and improve network binarization for broader adoption in AI compression, though it is incremental as it builds on existing binarization methods.
The paper tackles the problem of network binarization, which faces challenges like accuracy degradation and efficiency limitations in diverse real-world scenarios, by presenting BiBench, a benchmark that evaluates milestone algorithms and reveals key insights, such as the crucial impact of binarized operators and significant accuracy variations across tasks and architectures, with results showing promising efficiency on edge devices.
Network binarization emerges as one of the most promising compression approaches offering extraordinary computation and memory savings by minimizing the bit-width. However, recent research has shown that applying existing binarization algorithms to diverse tasks, architectures, and hardware in realistic scenarios is still not straightforward. Common challenges of binarization, such as accuracy degradation and efficiency limitation, suggest that its attributes are not fully understood. To close this gap, we present BiBench, a rigorously designed benchmark with in-depth analysis for network binarization. We first carefully scrutinize the requirements of binarization in the actual production and define evaluation tracks and metrics for a comprehensive and fair investigation. Then, we evaluate and analyze a series of milestone binarization algorithms that function at the operator level and with extensive influence. Our benchmark reveals that 1) the binarized operator has a crucial impact on the performance and deployability of binarized networks; 2) the accuracy of binarization varies significantly across different learning tasks and neural architectures; 3) binarization has demonstrated promising efficiency potential on edge devices despite the limited hardware support. The results and analysis also lead to a promising paradigm for accurate and efficient binarization. We believe that BiBench will contribute to the broader adoption of binarization and serve as a foundation for future research. The code for our BiBench is released https://github.com/htqin/BiBench .