Going Beyond Classification Accuracy Metrics in Model Compression
This work is significant for practitioners deploying compressed models on edge devices, as it improves alignment on critical metrics like fairness and explainability without altering the compression algorithm.
This paper addresses the problem of misalignment in metrics beyond accuracy (e.g., fairness, explainability) when models are compressed. The authors propose a novel multi-part loss function that reduces prediction mismatches between compressed and reference models by up to 4.1x overall, and up to 5.7x when the reference model is correct.
With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the model considerably without affecting the standard metrics such as top-1 accuracy. However, these pruning approaches tend to result in a significant mismatch in other metrics such as fairness across classes and explainability. To combat such misalignment, we propose a novel multi-part loss function inspired by the knowledge-distillation literature. Through extensive experiments, we demonstrate the effectiveness of our approach across different compression algorithms, architectures, tasks as well as datasets. In particular, we obtain up to $4.1\times$ reduction in the number of prediction mismatches between the compressed and reference models, and up to $5.7\times$ in cases where the reference model makes the correct prediction; all while making no changes to the compression algorithm, and minor modifications to the loss function. Furthermore, we demonstrate how inducing simple alignment between the predictions of the models naturally improves the alignment on other metrics including fairness and attributions. Our framework can thus serve as a simple plug-and-play component for compression algorithms in the future.