Minimizing Embedding Distortion for Robust Out-of-Distribution Performance
This addresses the challenge of robust adaptation for practitioners using foundational models, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of preserving foundational models' generalization capabilities during fine-tuning for downstream tasks by introducing a similarity loss that minimizes embedding distortion, resulting in significantly improved out-of-distribution performance while maintaining strong in-distribution performance on tasks like image classification and face recognition.
Foundational models, trained on vast and diverse datasets, have demonstrated remarkable capabilities in generalizing across different domains and distributions for various zero-shot tasks. Our work addresses the challenge of retaining these powerful generalization capabilities when adapting foundational models to specific downstream tasks through fine-tuning. To this end, we introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. By minimizing the distortion of fine-tuned embeddings from the pre-trained embeddings, our method strikes a balance between task-specific adaptation and preserving broad generalization abilities. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition, focusing on open-class and domain shift scenarios to assess out-of-distribution (OOD) performance. We demonstrate that this approach significantly improves OOD performance while maintaining strong in-distribution (ID) performance.