Improved statistical benchmarking of digital pathology models using pairwise frames evaluation
This addresses the challenge of data size and annotator variability in validating digital pathology models for medical applications, but it is incremental as it builds on existing benchmarking methods.
The paper tackled the problem of benchmarking digital pathology models by introducing nested pairwise frames evaluation, which compares model-pathologist agreement to inter-pathologist agreement, and demonstrated its application on an H&E-stained melanoma dataset for tasks like tissue classification and cell count prediction.
Nested pairwise frames is a method for relative benchmarking of cell or tissue digital pathology models against manual pathologist annotations on a set of sampled patches. At a high level, the method compares agreement between a candidate model and pathologist annotations with agreement among pathologists' annotations. This evaluation framework addresses fundamental issues of data size and annotator variability in using manual pathologist annotations as a source of ground truth for model validation. We implemented nested pairwise frames evaluation for tissue classification, cell classification, and cell count prediction tasks and show results for cell and tissue models deployed on an H&E-stained melanoma dataset.