From Authors to Reviewers: Leveraging Rankings to Improve Peer Review
This addresses the concern of declining review quality in ML conferences due to high submission volumes, but it is incremental as it builds on prior work by Su et al. (2025).
The paper tackles the problem of improving peer review quality in machine learning conferences by proposing an alternative approach that leverages ranking information from reviewers instead of authors, showing that this method often outperforms using author rankings alone and that combining both yields the most accurate evaluations in most scenarios.
This paper is a discussion of the 2025 JASA discussion paper by Su et al. (2025). We would like to congratulate the authors on conducting a comprehensive and insightful empirical investigation of the 2023 ICML ranking data. The review quality of machine learning (ML) conferences has become a big concern in recent years, due to the rapidly growing number of submitted manuscripts. In this discussion, we propose an approach alternative to Su et al. (2025) that leverages ranking information from reviewers rather than authors. We simulate review data that closely mimics the 2023 ICML conference submissions. Our results show that (i) incorporating ranking information from reviewers can significantly improve the evaluation of each paper's quality, often outperforming the use of ranking information from authors alone; and (ii) combining ranking information from both reviewers and authors yields the most accurate evaluation of submitted papers in most scenarios.