Towards Context-Aware Domain Generalization: Understanding the Benefits and Limits of Marginal Transfer Learning
This work addresses domain generalization for machine learning practitioners by offering theoretical and empirical insights into marginal transfer learning, though it is incremental as it builds on existing approaches.
The paper tackles the problem of improving deep learning model predictions in new domains by analyzing when context information helps, and provides criteria to identify favorable scenarios and detect unwarranted extrapolation in out-of-distribution domains.
In this work, we analyze the conditions under which information about the context of an input $X$ can improve the predictions of deep learning models in new domains. Following work in marginal transfer learning in Domain Generalization (DG), we formalize the notion of context as a permutation-invariant representation of a set of data points that originate from the same domain as the input itself. We offer a theoretical analysis of the conditions under which this approach can, in principle, yield benefits, and formulate two necessary criteria that can be easily verified in practice. Additionally, we contribute insights into the kind of distribution shifts for which the marginal transfer learning approach promises robustness. Empirical analysis shows that our criteria are effective in discerning both favorable and unfavorable scenarios. Finally, we demonstrate that we can reliably detect scenarios where a model is tasked with unwarranted extrapolation in out-of-distribution (OOD) domains, identifying potential failure cases. Consequently, we showcase a method to select between the most predictive and the most robust model, circumventing the well-known trade-off between predictive performance and robustness.