Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation
This work addresses the challenge of reducing expert annotation costs in soil science, but it is incremental as it builds on existing conformal prediction methods applied to a specific domain model.
The authors tackled the problem of efficiently annotating soil horizons by applying conformal prediction to SoilNet, a multimodal multitask model, to calibrate uncertainties and guide expert intervention in a simulated human-in-the-loop pipeline, resulting in more efficient annotation for regression tasks and comparable performance for classification tasks under the same budget.
Uncertainty quantification is essential in human-machine collaboration, as human agents tend to adjust their decisions based on the confidence of the machine counterpart. Reliably calibrated model uncertainties, hence, enable more effective collaboration, targeted expert intervention and more responsible usage of Machine Learning (ML) systems. Conformal prediction has become a well established model-agnostic framework for uncertainty calibration of ML models, offering statistically valid confidence estimates for both regression and classification tasks. In this work, we apply conformal prediction to $\textit{SoilNet}$, a multimodal multitask model for describing soil profiles. We design a simulated human-in-the-loop (HIL) annotation pipeline, where a limited budget for obtaining ground truth annotations from domain experts is available when model uncertainty is high. Our experiments show that conformalizing SoilNet leads to more efficient annotation in regression tasks and comparable performance scores in classification tasks under the same annotation budget when tested against its non-conformal counterpart. All code and experiments can be found in our repository: https://github.com/calgo-lab/BGR