We should avoid the assumption of data-generating probability distributions in social settings
This is a foundational critique that challenges a core assumption in ML, potentially affecting fairness and equity research, but it is incremental as it builds on existing philosophical critiques without introducing new empirical results.
The paper argues that the assumption of data-generating probability distributions in machine learning, especially in social contexts, is flawed and should be avoided, proposing alternative frameworks that focus on populations instead.
Machine Learning research, including work promoting fair or equitable algorithms, heavily relies on the concept of a data-generating probability distribution. The standard presumption is that since data points are 'sampled from' such a distribution, one can learn from observed data about this distribution and, thus, predict future data points which are also drawn from it. We argue, however, that such true probability distributions do not exist and should not be dealt with uncritically. We show that alternative frameworks focusing directly on relevant populations rather than abstract distributions are available and leave classical learning theory almost unchanged. Furthermore, we argue that the assumption of true probabilities or data-generating distributions can be misleading and obscure both the choices made and the goals pursued in machine learning practice. Based on these considerations, this position paper argues that, at least in social settings, machine learning work should avoid assuming data-generating probability distributions.