Expanding search in the space of empirical ML
This addresses a systemic issue for ML researchers and practitioners by proposing incremental changes to conference norms to improve empirical research.
The paper argues that current ML conferences overly emphasize algorithmic novelty, neglecting exploration of problem and data dimensions, and suggests revising reviewing criteria to better support empirical ML by incentivizing experimentation and synthesis independent of algorithmic innovation.
As researchers and practitioners of applied machine learning, we are given a set of requirements on the problem to be solved, the plausibly obtainable data, and the computational resources available. We aim to find (within those bounds) reliably useful combinations of problem, data, and algorithm. An emphasis on algorithmic or technical novelty in ML conference publications leads to exploration of one dimension of this space. Data collection and ML deployment at scale in industry settings offers an environment for exploring the others. Our conferences and reviewing criteria can better support empirical ML by soliciting and incentivizing experimentation and synthesis independent of algorithmic innovation.