Invariant Risk Minimization Is A Total Variation Model
This work provides a mathematical interpretation for IRM, potentially aiding researchers in understanding and improving out-of-distribution generalization methods, though it appears incremental as it builds on existing IRM concepts.
The authors identified that invariant risk minimization (IRM) is mathematically equivalent to a total variation model based on the L2 norm, and they proposed a new IRM framework using an L1 norm variant, which expands function classes and shows robust performance in denoising and invariant feature preservation. Experimental results indicate competitive performance on benchmark tasks.
Invariant risk minimization (IRM) is an arising approach to generalize invariant features to different environments in machine learning. While most related works focus on new IRM settings or new application scenarios, the mathematical essence of IRM remains to be properly explained. We verify that IRM is essentially a total variation based on $L^2$ norm (TV-$\ell_2$) of the learning risk with respect to the classifier variable. Moreover, we propose a novel IRM framework based on the TV-$\ell_1$ model. It not only expands the classes of functions that can be used as the learning risk and the feature extractor, but also has robust performance in denoising and invariant feature preservation based on the coarea formula. We also illustrate some requirements for IRM-TV-$\ell_1$ to achieve out-of-distribution generalization. Experimental results show that the proposed framework achieves competitive performance in several benchmark machine learning scenarios.