RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems
This addresses the bottleneck of evaluating reviewer expertise in peer review systems, particularly for AI/NLP, by providing an up-to-date benchmark and method, though it is incremental as it builds on existing embedding and ranking techniques.
The paper tackles the problem of outdated benchmarks and poor proxy signals for reviewer assignment in peer review by introducing LR-bench, a new benchmark with 1055 expert-annotated paper-reviewer-score annotations from 2024-2025, and proposes RATE, a reviewer-centric ranking framework that achieves state-of-the-art performance, outperforming strong baselines by a clear margin.
Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling matching each manuscript against a reviewer profile directly. Across LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/datasets/Gnociew/LR-bench, and a GitHub repository at https://github.com/Gnociew/RATE-Reviewer-Assign.