A Second-Order Perspective on Pruning at Initialization and Knowledge Transfer
This addresses the challenge of deploying computationally expensive pre-trained models when downstream tasks are unknown, offering a method for efficient task-specific adaptation without prior task data.
The paper tackles the problem of compressing pre-trained vision models for efficient deployment by investigating how data influences pruning-at-initialization, finding that pruning on one task retains zero-shot performance on unseen tasks and fine-tuning recovers performance on held-out tasks.
The widespread availability of pre-trained vision models has enabled numerous deep learning applications through their transferable representations. However, their computational and storage costs often limit practical deployment. Pruning-at-Initialization has emerged as a promising approach to compress models before training, enabling efficient task-specific adaptation. While conventional wisdom suggests that effective pruning requires task-specific data, this creates a challenge when downstream tasks are unknown in advance. In this paper, we investigate how data influences the pruning of pre-trained vision models. Surprisingly, pruning on one task retains the model's zero-shot performance also on unseen tasks. Furthermore, fine-tuning these pruned models not only improves performance on original seen tasks but can recover held-out tasks' performance. We attribute this phenomenon to the favorable loss landscapes induced by extensive pre-training on large-scale datasets.