The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions
This addresses the reliability of machine learning predictions for practitioners and researchers, highlighting risks from data uncertainties, but it is incremental as it builds on existing work on data bias and uncertainty.
The paper tackles the problem of how inaccuracies and biases in training datasets affect predictions by introducing the dataset multiplicity framework, showing that many test samples in real-world datasets are impacted under reasonable assumptions, with effects varying by demographic groups.
We introduce dataset multiplicity, a way to study how inaccuracies, uncertainty, and social bias in training datasets impact test-time predictions. The dataset multiplicity framework asks a counterfactual question of what the set of resultant models (and associated test-time predictions) would be if we could somehow access all hypothetical, unbiased versions of the dataset. We discuss how to use this framework to encapsulate various sources of uncertainty in datasets' factualness, including systemic social bias, data collection practices, and noisy labels or features. We show how to exactly analyze the impacts of dataset multiplicity for a specific model architecture and type of uncertainty: linear models with label errors. Our empirical analysis shows that real-world datasets, under reasonable assumptions, contain many test samples whose predictions are affected by dataset multiplicity. Furthermore, the choice of domain-specific dataset multiplicity definition determines what samples are affected, and whether different demographic groups are disparately impacted. Finally, we discuss implications of dataset multiplicity for machine learning practice and research, including considerations for when model outcomes should not be trusted.