Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
This addresses concerns about review quality for researchers and conference organizers, providing evidence against a common narrative.
The paper tackled the question of whether peer review quality is declining by developing a framework to analyze reviews from major AI/ML conferences (ICLR, NeurIPS, *ACL), finding no consistent decline in median review quality over time.
Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.