Dodgersort: Uncertainty-Aware VLM-Guided Human-in-the-Loop Pairwise Ranking
This work addresses the efficiency and reliability challenges in pairwise ranking for domains like medical imaging and aesthetics, offering incremental improvements over existing methods.
The paper tackles the problem of reducing the quadratic annotation cost in pairwise comparison labeling by proposing Dodgersort, which uses uncertainty-aware VLM-guided human-in-the-loop methods to achieve an 11-16% reduction in annotations while improving inter-rater reliability across visual ranking tasks.
Pairwise comparison labeling is emerging as it yields higher inter-rater reliability than conventional classification labeling, but exhaustive comparisons require quadratic cost. We propose Dodgersort, which leverages CLIP-based hierarchical pre-ordering, a neural ranking head and probabilistic ensemble (Elo, BTL, GP), epistemic--aleatoric uncertainty decomposition, and information-theoretic pair selection. It reduces human comparisons while improving the reliability of the rankings. In visual ranking tasks in medical imaging, historical dating, and aesthetics, Dodgersort achieves a 11--16\% annotation reduction while improving inter-rater reliability. Cross-domain ablations across four datasets show that neural adaptation and ensemble uncertainty are key to this gain. In FG-NET with ground-truth ages, the framework extracts 5--20$\times$ more ranking information per comparison than baselines, yielding Pareto-optimal accuracy--efficiency trade-offs.