GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study
This addresses resource constraints in peer review for academic communities, though it is incremental as it builds on existing AI tools.
The study investigated using GPT4 to assist peer-review for academic papers, finding that GPT-generated reviews achieved comparable helpfulness to human reviewers, with initial evidence supporting AI's effectiveness in this process.
In this pilot study, we investigate the use of GPT4 to assist in the peer-review process. Our key hypothesis was that GPT-generated reviews could achieve comparable helpfulness to human reviewers. By comparing reviews generated by both human reviewers and GPT models for academic papers submitted to a major machine learning conference, we provide initial evidence that artificial intelligence can contribute effectively to the peer-review process. We also perform robustness experiments with inserted errors to understand which parts of the paper the model tends to focus on. Our findings open new avenues for leveraging machine learning tools to address resource constraints in peer review. The results also shed light on potential enhancements to the review process and lay the groundwork for further research on scaling oversight in a domain where human-feedback is increasingly a scarce resource.