OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
This addresses the need for rapid, constructive feedback to improve manuscript quality before submission, though it is incremental as it builds on existing LLM fine-tuning techniques.
The authors tackled the problem of generating critical peer reviews for machine learning and AI papers by developing OpenReviewer, a specialized 8B parameter language model fine-tuned on 79,000 expert reviews, which produced more realistic and critical reviews than general-purpose LLMs like GPT-4 and Claude-3.5, closely matching human reviewer ratings in evaluations on 400 test papers.
We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.