Leveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging
This work addresses the reliability issue for biomedical applications by explicitly modeling rater disagreement, though it is incremental as it builds on existing ensemble and calibration techniques.
The paper tackled the problem of poorly calibrated confidence estimates in deep learning-based object detectors for microscopy imaging by introducing a method that leverages multi-rater annotations to train separate models and aggregate predictions, resulting in improved calibration performance while maintaining detection accuracy on a colorectal organoid dataset.
Deep learning-based object detectors have achieved impressive performance in microscopy imaging, yet their confidence estimates often lack calibration, limiting their reliability for biomedical applications. In this work, we introduce a new approach to improve model calibration by leveraging multi-rater annotations. We propose to train separate models on the annotations from single experts and aggregate their predictions to emulate consensus. This improves upon label sampling strategies, where models are trained on mixed annotations, and offers a more principled way to capture inter-rater variability. Experiments on a colorectal organoid dataset annotated by two experts demonstrate that our rater-specific ensemble strategy improves calibration performance while maintaining comparable detection accuracy. These findings suggest that explicitly modelling rater disagreement can lead to more trustworthy object detectors in biomedical imaging.