Pre-Training Transformers for Domain Adaptation
It improves model performance for domain adaptation in visual tasks, but is incremental as it applies an existing method to a new challenge.
The paper tackled unsupervised domain adaptation by using BeiT to transfer knowledge from source to target datasets, achieving first place in the ViSDA Challenge with 56.29% ACC and 69.79% AUROC.
The Visual Domain Adaptation Challenge 2021 called for unsupervised domain adaptation methods that could improve the performance of models by transferring the knowledge obtained from source datasets to out-of-distribution target datasets. In this paper, we utilize BeiT [1] and demonstrate its capability of capturing key attributes from source datasets and apply it to target datasets in a semi-supervised manner. Our method was able to outperform current state-of-the-art (SoTA) techniques and was able to achieve 1st place on the ViSDA Domain Adaptation Challenge with ACC of 56.29% and AUROC of 69.79%.