On the inductive biases of deep domain adaptation
This work addresses the theory-practice gap in domain adaptation for machine learning practitioners, offering a more realistic framework that is incremental in shifting focus from alignment to inductive biases.
The paper challenges the prevailing theory that domain alignment ensures low target risk in unsupervised domain adaptation, showing it is neither necessary nor sufficient, and instead identifies hidden inductive biases like pre-training and encoder design as key drivers of success, with meta-learned biases outperforming handcrafted ones in experiments.
Domain alignment is currently the most prevalent solution to unsupervised domain-adaptation tasks and are often being presented as minimizers of some theoretical upper-bounds on risk in the target domain. However, further works revealed severe inadequacies between theory and practice: we consolidate this analysis and confirm that imposing domain invariance on features is neither necessary nor sufficient to obtain low target risk. We instead argue that successful deep domain adaptation rely largely on hidden inductive biases found in the common practice, such as model pre-training or design of encoder architecture. We perform various ablation experiments on popular benchmarks and our own synthetic transfers to illustrate their role in prototypical situations. To conclude our analysis, we propose to meta-learn parametric inductive biases to solve specific transfers and show their superior performance over handcrafted heuristics.