Optimizing Resources for On-the-Fly Label Estimation with Multiple Unknown Medical Experts
This work addresses the need for seamless integration of annotation algorithms into medical screening workflows, offering a domain-specific solution for improving efficiency in expert-driven labeling tasks.
The paper tackles the problem of efficiently aggregating noisy expert annotations in medical screening by proposing an adaptive approach that supports real-time labeling without prior knowledge of experts or data, reducing expert queries by up to 50% while maintaining accuracy comparable to a non-adaptive baseline.
Accurate ground truth estimation in medical screening programs often relies on coalitions of experts and peer second opinions. Algorithms that efficiently aggregate noisy annotations can enhance screening workflows, particularly when data arrive continuously and expert proficiency is initially unknown. However, existing algorithms do not meet the requirements for seamless integration into screening pipelines. We therefore propose an adaptive approach for real-time annotation that (I) supports on-the-fly labeling of incoming data, (II) operates without prior knowledge of medical experts or pre-labeled data, and (III) dynamically queries additional experts based on the latent difficulty of each instance. The method incrementally gathers expert opinions until a confidence threshold is met, providing accurate labels with reduced annotation overhead. We evaluate our approach on three multi-annotator classification datasets across different modalities. Results show that our adaptive querying strategy reduces the number of expert queries by up to 50% while achieving accuracy comparable to a non-adaptive baseline. Our code is available at https://github.com/tbary/MEDICS