Approaching Peak Ground Truth
This addresses the issue of sub-optimal model predictions due to unreliable annotations for researchers and practitioners in fields like biomedical AI, though it appears incremental as it builds on existing reliability metrics.
The paper tackles the problem of evaluating machine learning models when reference annotations are subjective and unreliable, particularly in biomedical domains, by introducing the theoretical concept of Peak Ground Truth (PGT) and proposing a quantitative technique to approximate it using inter- and intra-rater reliability.
Machine learning models are typically evaluated by computing similarity with reference annotations and trained by maximizing similarity with such. Especially in the biomedical domain, annotations are subjective and suffer from low inter- and intra-rater reliability. Since annotations only reflect one interpretation of the real world, this can lead to sub-optimal predictions even though the model achieves high similarity scores. Here, the theoretical concept of PGT is introduced. PGT marks the point beyond which an increase in similarity with the \emph{reference annotation} stops translating to better RWMP. Additionally, a quantitative technique to approximate PGT by computing inter- and intra-rater reliability is proposed. Finally, four categories of PGT-aware strategies to evaluate and improve model performance are reviewed.