Adaptive Sharpness-Aware Pruning for Robust Sparse Networks
This addresses the challenge of deploying efficient and reliable models in real-world applications, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackled the problem of creating compact yet robust deep learning models by introducing Adaptive Sharpness-Aware Pruning (AdaSAP), which improved robust accuracy by up to +6% on ImageNet C and +4% on ImageNet V2 for image classification, and +4% on a corrupted Pascal VOC dataset for object detection.
Robustness and compactness are two essential attributes of deep learning models that are deployed in the real world. The goals of robustness and compactness may seem to be at odds, since robustness requires generalization across domains, while the process of compression exploits specificity in one domain. We introduce Adaptive Sharpness-Aware Pruning (AdaSAP), which unifies these goals through the lens of network sharpness. The AdaSAP method produces sparse networks that are robust to input variations which are unseen at training time. We achieve this by strategically incorporating weight perturbations in order to optimize the loss landscape. This allows the model to be both primed for pruning and regularized for improved robustness. AdaSAP improves the robust accuracy of pruned models on image classification by up to +6% on ImageNet C and +4% on ImageNet V2, and on object detection by +4% on a corrupted Pascal VOC dataset, over a wide range of compression ratios, pruning criteria, and network architectures, outperforming recent pruning art by large margins.