Do Public Datasets Assure Unbiased Comparisons for Registration Evaluation?
This work addresses potential biases in registration evaluation for clinicians, though it is incremental as it focuses on screening existing datasets rather than introducing new methods.
The study screened manually annotated landmarks in two neuroimaging datasets using variograms to assess annotation quality, finding that fiducial localization errors and suboptimal landmark distributions could bias registration evaluations.
With the increasing availability of new image registration approaches, an unbiased evaluation is becoming more needed so that clinicians can choose the most suitable approaches for their applications. Current evaluations typically use landmarks in manually annotated datasets. As a result, the quality of annotations is crucial for unbiased comparisons. Even though most data providers claim to have quality control over their datasets, an objective third-party screening can be reassuring for intended users. In this study, we use the variogram to screen the manually annotated landmarks in two datasets used to benchmark registration in image-guided neurosurgeries. The variogram provides an intuitive 2D representation of the spatial characteristics of annotated landmarks. Using variograms, we identified potentially problematic cases and had them examined by experienced radiologists. We found that (1) a small number of annotations may have fiducial localization errors; (2) the landmark distribution for some cases is not ideal to offer fair comparisons. If unresolved, both findings could incur bias in registration evaluation.