Calibration with Privacy in Peer Review
This addresses privacy concerns in peer review systems, but it is incremental as it builds on existing calibration methods with a theoretical focus on a limited scenario.
The paper tackles the problem of calibrating peer reviews while preserving reviewer privacy, establishing the tradeoff between privacy and utility and designing Pareto-optimal algorithms for a simplified model with two reviewers and two papers.
Reviewers in peer review are often miscalibrated: they may be strict, lenient, extreme, moderate, etc. A number of algorithms have previously been proposed to calibrate reviews. Such attempts of calibration can however leak sensitive information about which reviewer reviewed which paper. In this paper, we identify this problem of calibration with privacy, and provide a foundational building block to address it. Specifically, we present a theoretical study of this problem under a simplified-yet-challenging model involving two reviewers, two papers, and an MAP-computing adversary. Our main results establish the Pareto frontier of the tradeoff between privacy (preventing the adversary from inferring reviewer identity) and utility (accepting better papers), and design explicit computationally-efficient algorithms that we prove are Pareto optimal.