Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers
This work addresses the challenge of out-of-distribution detection for machine learning practitioners, offering a principled method to enhance testing accuracy, though it is incremental by building on existing conformal inference techniques.
The paper tackles the problem of testing whether new observations come from the same distribution as a reference set, developing novel conformal methods that re-weight p-values using known out-of-distribution data and achieve improved performance over standard approaches, as demonstrated through simulations and applications with concrete gains in power.
This paper develops novel conformal methods to test whether a new observation was sampled from the same distribution as a reference set. Blending inductive and transductive conformal inference in an innovative way, the described methods can re-weight standard conformal p-values based on dependent side information from known out-of-distribution data in a principled way, and can automatically take advantage of the most powerful model from any collection of one-class and binary classifiers. The solution can be implemented either through sample splitting or via a novel transductive cross-validation+ scheme which may also be useful in other applications of conformal inference, due to tighter guarantees compared to existing cross-validation approaches. After studying false discovery rate control and power within a multiple testing framework with several possible outliers, the proposed solution is shown to outperform standard conformal p-values through simulations as well as applications to image recognition and tabular data.