The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
This work addresses the need for automated support to ensure high-quality peer review feedback for authors in academic publishing, representing an incremental improvement with a new dataset and benchmarking.
The paper tackles the problem of automatically measuring the utility of peer review comments for authors by identifying four key aspects (Actionability, Grounding & Specificity, Verifiability, and Helpfulness) and introduces the RevUtil dataset with 1,430 human-labeled and 10k synthetically labeled comments. The result shows that fine-tuned models achieve agreement levels with humans comparable to or exceeding GPT-4o, while machine-generated reviews generally underperform human reviews on these aspects.
Providing constructive feedback to paper authors is a core component of peer review. With reviewers increasingly having less time to perform reviews, automated support systems are required to ensure high reviewing quality, thus making the feedback in reviews useful for authors. To this end, we identify four key aspects of review comments (individual points in weakness sections of reviews) that drive the utility for authors: Actionability, Grounding & Specificity, Verifiability, and Helpfulness. To enable evaluation and development of models assessing review comments, we introduce the RevUtil dataset. We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes. The synthetic data additionally contains rationales, i.e., explanations for the aspect score of a review comment. Employing the RevUtil dataset, we benchmark fine-tuned models for assessing review comments on these aspects and generating rationales. Our experiments demonstrate that these fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o. Our analysis further reveals that machine-generated reviews generally underperform human reviews on our four aspects.