Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation
This work addresses domain adaptation challenges for machine learning applications where data comes from multiple sources, offering theoretical insights and a practical algorithm, though it is incremental in nature.
The paper tackles multi-source domain adaptation (MDA) by deriving algorithm-dependent generalization bounds using mutual information and proposing a novel deep algorithm for joint distribution alignment, achieving comparable performance to state-of-the-art methods on target-shifted benchmarks with improved memory efficiency.
We use information-theoretic tools to derive a novel analysis of Multi-source Domain Adaptation (MDA) from the representation learning perspective. Concretely, we study joint distribution alignment for supervised MDA with few target labels and unsupervised MDA with pseudo labels, where the latter is relatively hard and less commonly studied. We further provide algorithm-dependent generalization bounds for these two settings, where the generalization is characterized by the mutual information between the parameters and the data. Then we propose a novel deep MDA algorithm, implicitly addressing the target shift through joint alignment. Finally, the mutual information bounds are extended to this algorithm providing a non-vacuous gradient-norm estimation. The proposed algorithm has comparable performance to the state-of-the-art on target-shifted MDA benchmark with improved memory efficiency.