A Unified Statistical Learning Model for Rankings and Scores with Application to Grant Panel Review
This work addresses a methodological gap for researchers and practitioners in fields like grant review, where both rankings and scores are commonly used, though it is incremental as it builds on existing ranking and score models.
The authors tackled the lack of a unified statistical model for handling both rankings and scores simultaneously, proposing the Mallows-Binomial model that combines these data types to quantify object quality and consensus, and demonstrated its application to grant panel review data with efficient parameter estimation and confidence-based ranking.
Rankings and scores are two common data types used by judges to express preferences and/or perceptions of quality in a collection of objects. Numerous models exist to study data of each type separately, but no unified statistical model captures both data types simultaneously without first performing data conversion. We propose the Mallows-Binomial model to close this gap, which combines a Mallows' $φ$ ranking model with Binomial score models through shared parameters that quantify object quality, a consensus ranking, and the level of consensus between judges. We propose an efficient tree-search algorithm to calculate the exact MLE of model parameters, study statistical properties of the model both analytically and through simulation, and apply our model to real data from an instance of grant panel review that collected both scores and partial rankings. Furthermore, we demonstrate how model outputs can be used to rank objects with confidence. The proposed model is shown to sensibly combine information from both scores and rankings to quantify object quality and measure consensus with appropriate levels of statistical uncertainty.