Stratify or Die: Rethinking Data Splits in Image Segmentation
This addresses evaluation bias in image segmentation for researchers and practitioners, particularly in small or imbalanced datasets, though it is incremental as it builds on existing stratification concepts.
The paper tackled the problem of biased evaluations in image segmentation due to unrepresentative test sets from random data splits, introducing Wasserstein-Driven Evolutionary Stratification (WDES) to optimize splits and showing it reduces performance variance and improves model evaluation across diverse tasks.
Random splitting of datasets in image segmentation often leads to unrepresentative test sets, resulting in biased evaluations and poor model generalization. While stratified sampling has proven effective for addressing label distribution imbalance in classification tasks, extending these ideas to segmentation remains challenging due to the multi-label structure and class imbalance typically present in such data. Building on existing stratification concepts, we introduce Iterative Pixel Stratification (IPS), a straightforward, label-aware sampling method tailored for segmentation tasks. Additionally, we present Wasserstein-Driven Evolutionary Stratification (WDES), a novel genetic algorithm designed to minimize the Wasserstein distance, thereby optimizing the similarity of label distributions across dataset splits. We prove that WDES is globally optimal given enough generations. Using newly proposed statistical heterogeneity metrics, we evaluate both methods against random sampling and find that WDES consistently produces more representative splits. Applying WDES across diverse segmentation tasks, including street scenes, medical imaging, and satellite imagery, leads to lower performance variance and improved model evaluation. Our results also highlight the particular value of WDES in handling small, imbalanced, and low-diversity datasets, where conventional splitting strategies are most prone to bias.