ConvBench: A Comprehensive Benchmark for 2D Convolution Primitive Evaluation
This provides a standardized tool for convolution algorithm designers to identify performance bottlenecks, though it is incremental as it builds on existing benchmarking efforts.
The paper tackles the problem of error-prone and limited benchmarking for convolution algorithms by introducing ConvBench, a comprehensive primitive-level benchmark that evaluates 9243 operations from 1097 real-world models, revealing that SConv outperforms Im2col-GEMM in 93.6% of cases but has a critical 79.5% average slowdown in its packing step.
Convolution is a compute-intensive operation placed at the heart of Convolution Neural Networks (CNNs). It has led to the development of many high-performance algorithms, such as Im2col-GEMM, Winograd, and Direct-Convolution. However, the comparison of different convolution algorithms is an error-prone task as it requires specific data layouts and system resources. Failure to address these requirements might lead to unwanted time penalties. Thus, considering all processing steps within convolution algorithms is essential to comprehensively evaluate and fairly compare their performance. Furthermore, most known convolution benchmarking adopts ad-hoc testing suites with limited coverage and handmade operations. This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms. It assesses 9243 convolution operations derived from 1097 real-world deep learning models, resulting in performance and execution breakdown graphs for a detailed evaluation. ConvBench capability is evaluated across the Sliced Convolution (SConv) algorithm. The experiments showed results faster than Im2col-GEMM in 93.6% of the convolutions. However, the use of ConvBench allowed the delving into the remaining 6.4% underperforming convolutions, uncovering a critical slowdown of 79.5% on average of SConv's packing step. This analysis underscores a potential source of optimization for SConv, opening up new paths for convolution designers to improve their algorithms.