Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance
This addresses robustness issues in fine-tuning for practitioners using pre-trained models, though it is incremental as it builds on existing methods.
The paper tackles the problem of fine-tuning zero-shot vision-language models to improve downstream performance without compromising robustness against distribution shifts, proposing Lipsum-FT which achieves superior results in experiments on DomainNet and ImageNet.
Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across a range of image classification tasks without requiring training on downstream data. Recent works have confirmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descriptions based on feature distortion theory and joint energy-based models. Subsequently, we propose a novel robust fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language modeling aspect of the vision-language pre-trained models. Extensive experiments conducted on distribution shift scenarios in DomainNet and ImageNet confirm the superiority of our proposed Lipsum-FT approach over existing robust fine-tuning methods.