Exploring the Design of Adaptation Protocols for Improved Generalization and Machine Learning Safety
This work addresses the problem of improving model safety and generalization for machine learning practitioners, but it is incremental as it builds on existing adaptation methods.
The paper investigates how different adaptation protocols for pretrained models affect out-of-distribution generalization and safety metrics, finding that protocols create trade-offs and that pairing data augmentation with protocols can mitigate these issues.
While directly fine-tuning (FT) large-scale, pretrained models on task-specific data is well-known to induce strong in-distribution task performance, recent works have demonstrated that different adaptation protocols, such as linear probing (LP) prior to FT, can improve out-of-distribution generalization. However, the design space of such adaptation protocols remains under-explored and the evaluation of such protocols has primarily focused on distribution shifts. Therefore, in this work, we evaluate common adaptation protocols across distributions shifts and machine learning safety metrics (e.g., anomaly detection, calibration, robustness to corruptions). We find that protocols induce disparate trade-offs that were not apparent from prior evaluation. Further, we demonstrate that appropriate pairing of data augmentation and protocol can substantially mitigate this trade-off. Finally, we hypothesize and empirically see that using hardness-promoting augmentations during LP and then FT with augmentations may be particularly effective for trade-off mitigation.