Position: Why We Must Rethink Empirical Research in Machine Learning
This addresses a foundational problem for the entire ML research community by highlighting methodological flaws that threaten progress.
The paper argues that current empirical research in machine learning is often mischaracterized as confirmatory, leading to non-replicable and unreliable results, and calls for a shift towards viewing it as exploratory to improve reliability.
We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.