Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation
This addresses fairness and discrimination concerns in machine learning applications by enabling efficient measurement and mitigation of predictive multiplicity, though it is an incremental improvement focused on computational efficiency.
The paper tackles the problem of predictive multiplicity in classification, where multiple models achieve similar performance but produce conflicting predictions, by proposing a dropout-based framework to efficiently explore the Rashomon set of almost-equally-optimal models, resulting in runtime speedups of up to 20x to 5000x compared to baselines.
Predictive multiplicity refers to the phenomenon in which classification tasks may admit multiple competing models that achieve almost-equally-optimal performance, yet generate conflicting outputs for individual samples. This presents significant concerns, as it can potentially result in systemic exclusion, inexplicable discrimination, and unfairness in practical applications. Measuring and mitigating predictive multiplicity, however, is computationally challenging due to the need to explore all such almost-equally-optimal models, known as the Rashomon set, in potentially huge hypothesis spaces. To address this challenge, we propose a novel framework that utilizes dropout techniques for exploring models in the Rashomon set. We provide rigorous theoretical derivations to connect the dropout parameters to properties of the Rashomon set, and empirically evaluate our framework through extensive experimentation. Numerical results show that our technique consistently outperforms baselines in terms of the effectiveness of predictive multiplicity metric estimation, with runtime speedup up to $20\times \sim 5000\times$. With efficient Rashomon set exploration and metric estimation, mitigation of predictive multiplicity is then achieved through dropout ensemble and model selection.