Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling
This work addresses uncertainty quantification for AI-assisted decision-making in high-stakes domains, but it is incremental as it empirically evaluates existing methods without introducing new techniques.
The study investigated the utility of conformal prediction sets for expressing uncertainty in AI-advised image labeling, finding that they offer some advantage in accuracy for out-of-distribution images, especially with small set sizes, but perform on par or worse for easy images.
As deep neural networks are more commonly deployed in high-stakes domains, their black-box nature makes uncertainty quantification challenging. We investigate the presentation of conformal prediction sets--a distribution-free class of methods for generating prediction sets with specified coverage--to express uncertainty in AI-advised decision-making. Through a large online experiment, we compare the utility of conformal prediction sets to displays of Top-1 and Top-k predictions for AI-advised image labeling. In a pre-registered analysis, we find that the utility of prediction sets for accuracy varies with the difficulty of the task: while they result in accuracy on par with or less than Top-1 and Top-k displays for easy images, prediction sets offer some advantage in assisting humans in labeling out-of-distribution (OOD) images in the setting that we studied, especially when the set size is small. Our results empirically pinpoint practical challenges of conformal prediction sets and provide implications on how to incorporate them for real-world decision-making.