Transferability of Representations Learned using Supervised Contrastive Learning Trained on a Multi-Domain Dataset
This addresses the problem of improving representation transferability across domains for machine learning practitioners, but it is incremental as it builds on existing contrastive learning methods.
The paper studied the transferability of representations learned using Supervised Contrastive Learning on a multi-domain dataset, finding that it performed 6.05% better on average than a cross-entropy baseline across 7 downstream datasets.
Contrastive learning has shown to learn better quality representations than models trained using cross-entropy loss. They also transfer better to downstream datasets from different domains. However, little work has been done to explore the transferability of representations learned using contrastive learning when trained on a multi-domain dataset. In this paper, a study has been conducted using the Supervised Contrastive Learning framework to learn representations from the multi-domain DomainNet dataset and then evaluate the transferability of the representations learned on other downstream datasets. The fixed feature linear evaluation protocol will be used to evaluate the transferability on 7 downstream datasets that were chosen across different domains. The results obtained are compared to a baseline model that was trained using the widely used cross-entropy loss. Empirical results from the experiments showed that on average, the Supervised Contrastive Learning model performed 6.05% better than the baseline model on the 7 downstream datasets. The findings suggest that Supervised Contrastive Learning models can potentially learn more robust representations that transfer better across domains than cross-entropy models when trained on a multi-domain dataset.