Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Density
This work addresses the issue of trustworthy machine learning by reducing reliance on non-robust features, which is incremental as it builds on existing regularization methods.
The paper tackles the problem of regulating model reliance on non-robust features by proposing a framework that attributes predictions to input features and regularizes the smoothness of the input marginal density, resulting in improved robustness against perturbations and mitigation of spurious correlations.
Trustworthy machine learning necessitates meticulous regulation of model reliance on non-robust features. We propose a framework to delineate and regulate such features by attributing model predictions to the input. Within our approach, robust feature attributions exhibit a certain consistency, while non-robust feature attributions are susceptible to fluctuations. This behavior allows identification of correlation between model reliance on non-robust features and smoothness of marginal density of the input samples. Hence, we uniquely regularize the gradients of the marginal density w.r.t. the input features for robustness. We also devise an efficient implementation of our regularization to address the potential numerical instability of the underlying optimization process. Moreover, we analytically reveal that, as opposed to our marginal density smoothing, the prevalent input gradient regularization smoothens conditional or joint density of the input, which can cause limited robustness. Our experiments validate the effectiveness of the proposed method, providing clear evidence of its capability to address the feature leakage problem and mitigate spurious correlations. Extensive results further establish that our technique enables the model to exhibit robustness against perturbations in pixel values, input gradients, and density.