Breaking Correlation Shift via Conditional Invariant Regularizer
This addresses generalization issues for machine learning models when spurious correlations vary between training and test data, but it appears incremental as it builds on existing conditional independence ideas.
The paper tackles the problem of out-of-distribution (OOD) generalization under correlation shift by proposing a conditional invariant regularizer based on a metric called Conditional Spurious Variation (CSV), which controls OOD error and is optimized via a nonconvex-concave mini-max algorithm with proven convergence, showing efficacy in empirical results.
Recently, generalization on out-of-distribution (OOD) data with correlation shift has attracted great attentions. The correlation shift is caused by the spurious attributes that correlate to the class label, as the correlation between them may vary in training and test data. For such a problem, we show that given the class label, the models that are conditionally independent of spurious attributes are OOD generalizable. Based on this, a metric Conditional Spurious Variation (CSV) which controls the OOD generalization error, is proposed to measure such conditional independence. To improve the OOD generalization, we regularize the training process with the proposed CSV. Under mild assumptions, our training objective can be formulated as a nonconvex-concave mini-max problem. An algorithm with a provable convergence rate is proposed to solve the problem. Extensive empirical results verify our algorithm's efficacy in improving OOD generalization.