Effectiveness of Anonymization in Double-Blind Review
This research addresses the reliability of double-blind review for academic conferences, providing empirical evidence that anonymization is generally effective, though it is incremental in evaluating existing practices.
The study investigated the effectiveness of anonymization in double-blind review at three conferences (ASE 2016, OOPSLA 2016, PLDI 2016) by assessing reviewers' ability to guess author identities, finding that 74%-90% of reviews contained no correct guesses and that expert reviewers were more likely to attempt guesses but not more likely to be correct.
Double-blind review relies on the authors' ability and willingness to effectively anonymize their submissions. We explore anonymization effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if they can guess author identities. We find that 74%-90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper's topic are more likely to attempt to guess, but no more likely to guess correctly. We present our findings, summarize the PC chairs' comments about administering double-blind review, discuss the advantages and disadvantages of revealing author identities part of the way through the process, and conclude by advocating for the continued use of double-blind review.