Avoiding a Tragedy of the Commons in the Peer Review Process
This addresses sustainability issues in peer review for the ML community, but it is incremental as it builds on existing ideas with specific proposals.
The paper tackles the problem of declining review quality in machine learning peer review due to rapid growth, proposing a rubric for objective standards and financial incentives to avoid a tragedy of the commons.
Peer review is the foundation of scientific publication, and the task of reviewing has long been seen as a cornerstone of professional service. However, the massive growth in the field of machine learning has put this community benefit under stress, threatening both the sustainability of an effective review process and the overall progress of the field. In this position paper, we argue that a tragedy of the commons outcome may be avoided by emphasizing the professional aspects of this service. In particular, we propose a rubric to hold reviewers to an objective standard for review quality. In turn, we also propose that reviewers be given appropriate incentive. As one possible such incentive, we explore the idea of financial compensation on a per-review basis. We suggest reasonable funding models and thoughts on long term effects.