Making learning more transparent using conformalized performance prediction
This work addresses the need for more transparent performance guarantees in machine learning, though it appears incremental as it builds on existing conformal prediction frameworks.
The authors tackled the problem of providing transparent and accurate performance guarantees for machine learning algorithms by extending conformal inference to predict future performance on unseen training sets, achieving valid and well-calibrated predictions as demonstrated in empirical examples.
In this work, we study some novel applications of conformal inference techniques to the problem of providing machine learning procedures with more transparent, accurate, and practical performance guarantees. We provide a natural extension of the traditional conformal prediction framework, done in such a way that we can make valid and well-calibrated predictive statements about the future performance of arbitrary learning algorithms, when passed an as-yet unseen training set. In addition, we include some nascent empirical examples to illustrate potential applications.