Pick up the PACE: Fast and Simple Domain Adaptation via Ensemble Pseudo-Labeling
This work addresses the computational inefficiency and hyperparameter sensitivity of state-of-the-art domain adaptation methods, offering a more practical solution for researchers and practitioners.
The paper tackles the problem of domain adaptation in deep learning by proposing PACE, a fast and simple method that combines domain alignment, pseudo-labeling, and ensembling, achieving 5-10% accuracy improvements on benchmark tasks and reducing training and hyperparameter tuning times by 82% and 97%, respectively.
Domain Adaptation (DA) has received widespread attention from deep learning researchers in recent years because of its potential to improve test accuracy with out-of-distribution labeled data. Most state-of-the-art DA algorithms require an extensive amount of hyperparameter tuning and are computationally intensive due to the large batch sizes required. In this work, we propose a fast and simple DA method consisting of three stages: (1) domain alignment by covariance matching, (2) pseudo-labeling, and (3) ensembling. We call this method $\textbf{PACE}$, for $\textbf{P}$seudo-labels, $\textbf{A}$lignment of $\textbf{C}$ovariances, and $\textbf{E}$nsembles. PACE is trained on top of fixed features extracted from an ensemble of modern pretrained backbones. PACE exceeds previous state-of-the-art by $\textbf{5 - 10 \%}$ on most benchmark adaptation tasks without training a neural network. PACE reduces training time and hyperparameter tuning time by $82\%$ and $97\%$, respectively, when compared to state-of-the-art DA methods. Code is released here: https://github.com/Chris210634/PACE-Domain-Adaptation