Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective
This work addresses the difficulty and time-consuming nature of final decision-making for program chairs in academic peer review, though it is incremental as it builds on existing ranking and preference learning methods.
The paper tackles the problem of decision-making in scholarly peer review by formalizing it as a paper ranking task, proposing a multi-faceted evaluation framework and a preference-learning approach that uses both review texts and scores. Experiments on ACL 2018 data show this approach outperforms baselines and prior work, emphasizing the value of combining text and scores for ranking.
Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.