Model-Based Domain Generalization
This addresses the problem of catastrophic failure in deep learning on out-of-distribution data for researchers and practitioners in machine learning, with incremental contributions building on existing domain generalization methods.
The paper tackles the domain generalization problem by proposing a model-based approach that formulates it as a constrained statistical learning problem and develops an algorithm with unconstrained relaxations using nonconvex duality theory. The result shows improvements of up to 30 percentage points over state-of-the-art baselines on benchmarks like ColoredMNIST and Camelyon17-WILDS.
Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30 percentage points over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.