Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference
This addresses the challenge of reliable classification with uncertainty estimates in scientific domains like biology and astroparticle physics, where mechanistic models exist but data distributions shift, offering a domain adaptation solution for pre-trained classifiers.
The paper tackles the problem of classification under distributional shifts in both labels and latent nuisance parameters, known as generalized label shift (GLS), by proposing a method that casts classification as a hypothesis testing problem to estimate ROC curves across nuisance parameters, resulting in robust uncertainty quantification and valid prediction sets.
An open scientific challenge is how to classify events with reliable measures of uncertainty, when we have a mechanistic model of the data-generating process but the distribution over both labels and latent nuisance parameters is different between train and target data. We refer to this type of distributional shift as generalized label shift (GLS). Direct classification using observed data $\mathbf{X}$ as covariates leads to biased predictions and invalid uncertainty estimates of labels $Y$. We overcome these biases by proposing a new method for robust uncertainty quantification that casts classification as a hypothesis testing problem under nuisance parameters. The key idea is to estimate the classifier's receiver operating characteristic (ROC) across the entire nuisance parameter space, which allows us to devise cutoffs that are invariant under GLS. Our method effectively endows a pre-trained classifier with domain adaptation capabilities and returns valid prediction sets while maintaining high power. We demonstrate its performance on two challenging scientific problems in biology and astroparticle physics with data from realistic mechanistic models.