Impact of Leakage on Data Harmonization in Machine Learning Pipelines in Class Imbalance Across Sites
This addresses data harmonization issues in biomedical ML pipelines, particularly for researchers dealing with imbalanced multi-site datasets, though it is incremental as it builds on existing harmonization techniques.
The study tackled the problem of data leakage in ComBat-based harmonization methods when class balance varies across sites, and found that their proposed PrettYharmonize method avoids leakage while achieving comparable performance in real-world MRI and clinical data.
Machine learning (ML) models benefit from large datasets. Collecting data in biomedical domains is costly and challenging, hence, combining datasets has become a common practice. However, datasets obtained under different conditions could present undesired site-specific variability. Data harmonization methods aim to remove site-specific variance while retaining biologically relevant information. This study evaluates the effectiveness of popularly used ComBat-based methods for harmonizing data in scenarios where the class balance is not equal across sites. We find that these methods struggle with data leakage issues. To overcome this problem, we propose a novel approach PrettYharmonize, designed to harmonize data by pretending the target labels. We validate our approach using controlled datasets designed to benchmark the utility of harmonization. Finally, using real-world MRI and clinical data, we compare leakage-prone methods with PrettYharmonize and show that it achieves comparable performance while avoiding data leakage, particularly in site-target-dependence scenarios.