Zero-shot reasoning for simulating scholarly peer-review
This addresses the problem of unmanageable submission volumes and unregulated AI in scholarly publishing for the scientific community and publishing strategists, offering a scalable tool for governance, though it appears incremental as a simulation framework building on existing AI methods.
The paper tackles the crisis in scholarly publishing by developing a deterministic simulation framework that provides the first stable, evidence-based standard for evaluating AI-generated peer review reports, demonstrating reliability through analysis of 352 reports with concrete metrics like >50% 'Revise' decisions and 45% 'Reject' rates in Health Sciences.
The scholarly publishing ecosystem faces a dual crisis of unmanageable submission volumes and unregulated AI, creating an urgent need for new governance models to safeguard scientific integrity. The traditional human-only peer review regime lacks a scalable, objective benchmark, making editorial processes opaque and difficult to audit. Here we investigate a deterministic simulation framework that provides the first stable, evidence-based standard for evaluating AI-generated peer review reports. Analyzing 352 peer-review simulation reports, we identify consistent system state indicators that demonstrate its reliability. First, the system is able to simulate calibrated editorial judgment, with 'Revise' decisions consistently forming the majority outcome (>50%) across all disciplines, while 'Reject' rates dynamically adapt to field-specific norms, rising to 45% in Health Sciences. Second, it maintains unwavering procedural integrity, enforcing a stable 29% evidence-anchoring compliance rate that remains invariant across diverse review tasks and scientific domains. These findings demonstrate a system that is predictably rule-bound, mitigating the stochasticity of generative AI. For the scientific community, this provides a transparent tool to ensure fairness; for publishing strategists, it offers a scalable instrument for auditing workflows, managing integrity risks, and implementing evidence-based governance. The framework repositions AI as an essential component of institutional accountability, providing the critical infrastructure to maintain trust in scholarly communication.