Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences
This addresses the need for transparency and evidence-based improvements in peer review for AI researchers and conference organizers, though it is incremental as it builds on existing data collection and analysis methods.
The paper tackles the problem of a strained peer-review system in AI conferences by introducing Paper Copilot, a system that creates digital archives of peer reviews and provides an open dataset, enabling large-scale empirical analysis of ICLR reviews over multiple years to study peer-review evolution.
The rapid growth of AI conferences is straining an already fragile peer-review system, leading to heavy reviewer workloads, expertise mismatches, inconsistent evaluation standards, superficial or templated reviews, and limited accountability under compressed timelines. In response, conference organizers have introduced new policies and interventions to preserve review standards. Yet these ad-hoc changes often create further concerns and confusion about the review process, leaving how papers are ultimately accepted - and how practices evolve across years - largely opaque. We present Paper Copilot, a system that creates durable digital archives of peer reviews across a wide range of computer-science venues, an open dataset that enables researchers to study peer review at scale, and a large-scale empirical analysis of ICLR reviews spanning multiple years. By releasing both the infrastructure and the dataset, Paper Copilot supports reproducible research on the evolution of peer review. We hope these resources help the community track changes, diagnose failure modes, and inform evidence-based improvements toward a more robust, transparent, and reliable peer-review system.