Is Large-Scale Pretraining the Secret to Good Domain Generalization?
This work addresses a critical issue in domain generalization by revealing limitations of current methods, highlighting the need for approaches that generalize beyond pretraining alignment, which is incremental but important for the field.
The paper investigates whether improvements in domain generalization (DG) benchmarks are due to better methods or stronger pretraining, finding that current DG methods perform well only on data similar to pretraining data and struggle otherwise. They introduce the Alignment Hypothesis, which links DG performance to the alignment of image and text embeddings, and confirm it experimentally.
Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.