Domain Generalization without Excess Empirical Risk
This work addresses a failure mode in domain generalization for machine learning practitioners, offering an incremental improvement to existing penalty-based methods.
The paper tackles the problem of excess empirical risk in domain generalization by proposing an approach that minimizes the penalty under the constraint of optimal empirical risk, ensuring in-distribution performance is not impaired. The method, applied to three existing techniques, shows significant improvements.
Given data from diverse sets of distinct distributions, domain generalization aims to learn models that generalize to unseen distributions. A common approach is designing a data-driven surrogate penalty to capture generalization and minimize the empirical risk jointly with the penalty. We argue that a significant failure mode of this recipe is an excess risk due to an erroneous penalty or hardness in joint optimization. We present an approach that eliminates this problem. Instead of jointly minimizing empirical risk with the penalty, we minimize the penalty under the constraint of optimality of the empirical risk. This change guarantees that the domain generalization penalty cannot impair optimization of the empirical risk, i.e., in-distribution performance. To solve the proposed optimization problem, we demonstrate an exciting connection to rate-distortion theory and utilize its tools to design an efficient method. Our approach can be applied to any penalty-based domain generalization method, and we demonstrate its effectiveness by applying it to three examplar methods from the literature, showing significant improvements.