Supervised Contrastive Learning with Heterogeneous Similarity for Distribution Shifts
This addresses the problem of model robustness under distribution shifts for machine learning practitioners, but it is incremental as it builds on existing contrastive learning techniques.
The paper tackles performance degradation in models due to distribution shifts by proposing a new regularization method using supervised contrastive learning with a heterogeneous similarity measure, showing advantages over existing methods on benchmark datasets.
Distribution shifts are problems where the distribution of data changes between training and testing, which can significantly degrade the performance of a model deployed in the real world. Recent studies suggest that one reason for the degradation is a type of overfitting, and that proper regularization can mitigate the degradation, especially when using highly representative models such as neural networks. In this paper, we propose a new regularization using the supervised contrastive learning to prevent such overfitting and to train models that do not degrade their performance under the distribution shifts. We extend the cosine similarity in contrastive loss to a more general similarity measure and propose to use different parameters in the measure when comparing a sample to a positive or negative example, which is analytically shown to act as a kind of margin in contrastive loss. Experiments on benchmark datasets that emulate distribution shifts, including subpopulation shift and domain generalization, demonstrate the advantage of the proposed method over existing regularization methods.