Evaluation Metrics for DNNs Compression
This addresses the problem of inconsistent evaluation for researchers and practitioners in deep learning compression, though it is incremental as it builds on existing metrics.
The paper tackles the lack of standardized evaluation metrics for neural network compression by reviewing existing metrics and introducing a framework called NetZIP with two novel metrics, CHATS and OCS, demonstrated through case studies on PC and Raspberry Pi 4 for object classification and detection.
There is a lot of ongoing research effort into developing different techniques for neural networks compression. However, the community lacks standardised evaluation metrics, which are key to identifying the most suitable compression technique for different applications. This paper reviews existing neural network compression evaluation metrics and implements them into a standardisation framework called NetZIP. We introduce two novel metrics to cover existing gaps of evaluation in the literature: 1) Compression and Hardware Agnostic Theoretical Speed (CHATS) and 2) Overall Compression Success (OCS). We demonstrate the use of NetZIP using two case studies on two different hardware platforms (a PC and a Raspberry Pi 4) focusing on object classification and object detection.