The $s$-value: evaluating stability with respect to distributional shifts
This addresses the challenge of transferability across datasets for statisticians and data scientists, though it appears incremental as it builds on existing divergence-based methods.
The paper tackles the problem of quantifying uncertainty due to distributional shifts in statistical parameters, proposing a measure called the $s$-value that evaluates sensitivity to perturbations within a Kullback-Leibler divergence ball, and shows it can improve estimation accuracy under shifted distributions on real data.
Common statistical measures of uncertainty such as $p$-values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect the type of shifts a parameter is sensitive to. We discuss how such knowledge can inform data collection for improved estimation of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.