Stability Evaluation via Distributional Perturbation Analysis
This addresses the need for reliable model deployment under data corruptions and sub-population shifts, offering a practical tool for stability comparison and improvement, though it is incremental as it builds on existing perturbation and optimal transport concepts.
The paper tackles the problem of learning models deteriorating in out-of-sample environments by proposing a stability evaluation criterion based on distributional perturbations, using optimal transport discrepancy to quantify minimal perturbations needed for risk deterioration, and validates it empirically across real-world applications.
The performance of learning models often deteriorates when deployed in out-of-sample environments. To ensure reliable deployment, we propose a stability evaluation criterion based on distributional perturbations. Conceptually, our stability evaluation criterion is defined as the minimal perturbation required on our observed dataset to induce a prescribed deterioration in risk evaluation. In this paper, we utilize the optimal transport (OT) discrepancy with moment constraints on the \textit{(sample, density)} space to quantify this perturbation. Therefore, our stability evaluation criterion can address both \emph{data corruptions} and \emph{sub-population shifts} -- the two most common types of distribution shifts in real-world scenarios. To further realize practical benefits, we present a series of tractable convex formulations and computational methods tailored to different classes of loss functions. The key technical tool to achieve this is the strong duality theorem provided in this paper. Empirically, we validate the practical utility of our stability evaluation criterion across a host of real-world applications. These empirical studies showcase the criterion's ability not only to compare the stability of different learning models and features but also to provide valuable guidelines and strategies to further improve models.