Doctor Rashomon and the UNIVERSE of Madness: Variable Importance with Unobserved Confounding and the Rashomon Effect
This addresses a critical issue for researchers and practitioners in fields like healthcare and finance who rely on variable importance for hypothesis generation and feature selection, offering a robust method to handle missing data and model variability.
The paper tackles the problem of variable importance estimation being unreliable due to unobserved confounders and the Rashomon effect, and introduces UNIVERSE to produce bounds on true variable importance with theoretical guarantees and strong performance in simulations and a credit risk task.
Variable importance (VI) methods are often used for hypothesis generation, feature selection, and scientific validation. In the standard VI pipeline, an analyst estimates VI for a single predictive model with only the observed features. However, the importance of a feature depends heavily on which other variables are included in the model, and essential variables are often omitted from observational datasets. Moreover, the VI estimated for one model is often not the same as the VI estimated for another equally-good model - a phenomenon known as the Rashomon Effect. We address these gaps by introducing UNobservables and Inference for Variable importancE using Rashomon SEts (UNIVERSE). Our approach adapts Rashomon sets - the sets of near-optimal models in a dataset - to produce bounds on the true VI even with missing features. We theoretically guarantee the robustness of our approach, show strong performance on semi-synthetic simulations, and demonstrate its utility in a credit risk task.