A Simple Way to Deal with Cherry-picking
This addresses the issue of false discoveries in ML research, which can mislead the scientific community, though it is an incremental solution focused on verification rather than prevention.
The paper tackles the problem of selection bias in machine learning, where researchers can falsely report algorithmic improvements by cherry-picking results from many configurations, and proposes a post-reporting verification method to detect such bias, supported by experiments on synthetic and real-world datasets.
Statistical hypothesis testing serves as statistical evidence for scientific innovation. However, if the reported results are intentionally biased, hypothesis testing no longer controls the rate of false discovery. In particular, we study such selection bias in machine learning models where the reporter is motivated to promote an algorithmic innovation. When the number of possible configurations (e.g., datasets) is large, we show that the reporter can falsely report an innovation even if there is no improvement at all. We propose a `post-reporting' solution to this issue where the bias of the reported results is verified by another set of results. The theoretical findings are supported by experimental results with synthetic and real-world datasets.