Least Square Calibration for Peer Review
This addresses calibration issues in academic conferences, offering a flexible algorithmic alternative to manual methods, though it appears incremental as it builds on prior work on scoring functions.
The paper tackles miscalibration in peer review systems by proposing a least square calibration (LSC) framework to select top candidates from ratings, which provably achieves perfect calibration under noiseless linear conditions and shows competitive performance with broader scoring functions and noise.
Peer review systems such as conference paper review often suffer from the issue of miscalibration. Previous works on peer review calibration usually only use the ordinal information or assume simplistic reviewer scoring functions such as linear functions. In practice, applications like academic conferences often rely on manual methods, such as open discussions, to mitigate miscalibration. It remains an important question to develop algorithms that can handle different types of miscalibrations based on available prior knowledge. In this paper, we propose a flexible framework, namely least square calibration (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions, yet also provides competitive calibration results when the scoring function is from broader classes beyond linear functions and with arbitrary noise. On our synthetic dataset, we empirically demonstrate that our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.