Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
For the ML/AI peer review community, this work proposes a novel approach to improve review quality, but the rejoinder is incremental, refining existing ideas rather than introducing new results.
The authors respond to critiques of their ICML 2023 ranking experiment, which proposed using author self-assessments and the Isotonic Mechanism to improve peer review. They address statistical formulation, equity concerns, integration of additional signals, and the role of generative AI.
This article is the rejoinder to ``The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review,'' to appear in the Journal of the American Statistical Association with discussion. To address the practical and theoretical points raised by the discussants, we organize our response around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.