Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions
This addresses a critical security flaw in peer review for ML/AI conferences, potentially affecting fairness and integrity, though it is incremental as it builds on known collusion issues.
The paper demonstrates that colluding reviewers and authors can exploit text-matching in ML/AI conference reviewer assignments to get assigned target papers, even without bid manipulation, highlighting specific vulnerabilities and suggesting improvements.
In the peer review process of top-tier machine learning (ML) and artificial intelligence (AI) conferences, reviewers are assigned to papers through automated methods. These assignment algorithms consider two main factors: (1) reviewers' expressed interests indicated by their bids for papers, and (2) reviewers' domain expertise inferred from the similarity between the text of their previously published papers and the submitted manuscripts. A significant challenge these conferences face is the existence of collusion rings, where groups of researchers manipulate the assignment process to review each other's papers, providing positive evaluations regardless of their actual quality. Most efforts to combat collusion rings have focused on preventing bid manipulation, under the assumption that the text similarity component is secure. In this paper, we demonstrate that even in the absence of bidding, colluding reviewers and authors can exploit the machine learning based text-matching component of reviewer assignment used at top ML/AI venues to get assigned their target paper. We also highlight specific vulnerabilities within this system and offer suggestions to enhance its robustness.