Risk Variance Penalization
This work addresses theoretical gaps in domain generalization methods for machine learning, but it is incremental as it builds on prior techniques.
The paper tackles the problem of out-of-distribution generalization by proposing Risk Variance Penalization (RVP), a method that modifies an existing regularization approach to provide theoretical justifications and improve robustness, with experimental validation under specific conditions.
The key of the out-of-distribution (OOD) generalization is to generalize invariance from training domains to target domains. The variance risk extrapolation (V-REx) is a practical OOD method, which depends on a domain-level regularization but lacks theoretical verifications about its motivation and utility. This article provides theoretical insights into V-REx by studying a variance-based regularizer. We propose Risk Variance Penalization (RVP), which slightly changes the regularization of V-REx but addresses the theory concerns about V-REx. We provide theoretical explanations and a theory-inspired tuning scheme for the regularization parameter of RVP. Our results point out that RVP discovers a robust predictor. Finally, we experimentally show that the proposed regularizer can find an invariant predictor under certain conditions.