CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
This addresses the problem of high deployment costs for foundation models in visual tasks, offering a practical tool for efficient adaptation, though it is incremental as it builds on existing compression and fine-tuning methods.
The paper tackles the performance degradation issue in post-training compression of foundation models by proposing CrAFT, a fine-tuning framework that uses sharpness minimization to enable effective compression, achieving significant boosts in one-shot pruning and quantization with minimal training overhead.
Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractical. Post-training compression techniques such as pruning and quantization can help lower deployment costs. Unfortunately, the resulting performance degradation limits the usability and benefits of such techniques. To close this performance gap, we propose CrAFT, a simple fine-tuning framework that enables effective post-training network compression. In CrAFT, users simply employ the default fine-tuning schedule along with sharpness minimization objective, simultaneously facilitating task adaptation and compression-friendliness. Contrary to the conventional sharpness minimization techniques, which are applied during pretraining, the CrAFT approach adds negligible training overhead as fine-tuning is done in under a couple of minutes or hours with a single GPU. The effectiveness of CrAFT, which is a general-purpose tool that can significantly boost one-shot pruning and post-training quantization, is demonstrated on both convolution-based and attention-based vision foundation models on a variety of target tasks. The code will be made publicly available.