Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
This highlights a critical issue for researchers using multi-site neuroimaging data with machine learning, as current harmonization methods fail to remove scanner-specific bias, making it incremental in addressing data quality challenges.
The study tackled the problem of scanner effects in multi-site neuroimaging data, finding that even after pre-processing, a classifier could distinguish data origin with high accuracy, leading to biased performance estimates and poor generalization in tasks like sex classification.
This is an empirical study to investigate the impact of scanner effects when using machine learning on multi-site neuroimaging data. We utilize structural T1-weighted brain MRI obtained from two different studies, Cam-CAN and UK Biobank. For the purpose of our investigation, we construct a dataset consisting of brain scans from 592 age- and sex-matched individuals, 296 subjects from each original study. Our results demonstrate that even after careful pre-processing with state-of-the-art neuroimaging pipelines a classifier can easily distinguish between the origin of the data with very high accuracy. Our analysis on the example application of sex classification suggests that current approaches to harmonize data are unable to remove scanner-specific bias leading to overly optimistic performance estimates and poor generalization. We conclude that multi-site data harmonization remains an open challenge and particular care needs to be taken when using such data with advanced machine learning methods for predictive modelling.