The Role of Embedding Complexity in Domain-invariant Representations
This work addresses domain adaptation for machine learning practitioners by providing a theoretical and empirical analysis of embedding complexity, though it appears incremental as it builds on existing domain-invariant representation methods.
The paper investigates how embedding complexity affects generalization in unsupervised domain adaptation, showing it influences an upper bound on target risk and proposing a strategy that mitigates sensitivity to this complexity, achieving performance comparable to or better than best layer-dependent tradeoffs.
Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff.