On Invariance Penalties for Risk Minimization
This work addresses domain generalization for machine learning models by improving invariance penalties, though it is incremental as it builds on existing IRM principles.
The paper tackles the problem of domain generalization by addressing flaws in the Invariant Risk Minimization (IRM) invariance penalty, which can be arbitrarily small for non-invariant representations, and proposes an alternative penalty based on the Gramian matrix eigenvalues, demonstrating effectiveness on DomainBed and InvarianceUnitTest benchmarks.
The Invariant Risk Minimization (IRM) principle was first proposed by Arjovsky et al. [2019] to address the domain generalization problem by leveraging data heterogeneity from differing experimental conditions. Specifically, IRM seeks to find a data representation under which an optimal classifier remains invariant across all domains. Despite the conceptual appeal of IRM, the effectiveness of the originally proposed invariance penalty has recently been brought into question. In particular, there exists counterexamples for which that invariance penalty can be arbitrarily small for non-invariant data representations. We propose an alternative invariance penalty by revisiting the Gramian matrix of the data representation. We discuss the role of its eigenvalues in the relationship between the risk and the invariance penalty, and demonstrate that it is ill-conditioned for said counterexamples. The proposed approach is guaranteed to recover an invariant representation for linear settings under mild non-degeneracy conditions. Its effectiveness is substantiated by experiments on DomainBed and InvarianceUnitTest, two extensive test beds for domain generalization.