Investigating Data Usage for Inductive Conformal Predictors
This addresses a data efficiency problem for researchers and practitioners using inductive conformal predictors, but it is incremental as it builds on existing methods.
The study investigated how to efficiently divide limited development data into training, calibration, and test sets for inductive conformal predictors, concluding with practical recommendations for data usage.
Inductive conformal predictors (ICPs) are algorithms that are able to generate prediction sets, instead of point predictions, which are valid at a user-defined confidence level, only assuming exchangeability. These algorithms are useful for reliable machine learning and are increasing in popularity. The ICP development process involves dividing development data into three parts: training, calibration and test. With access to limited or expensive development data, it is an open question regarding the most efficient way to divide the data. This study provides several experiments to explore this question and consider the case for allowing overlap of examples between training and calibration sets. Conclusions are drawn that will be of value to academics and practitioners planning to use ICPs.