Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification
This work addresses a methodological gap in evaluating multi-label ranking models, which is incremental but important for improving consistency in visual tasks.
The paper tackles the inconsistency of models across different ranking-based measures in multi-label classification by proposing a new measure called Top-K Pairwise Ranking (TKPR) and an empirical surrogate risk minimization framework. The framework achieves convex surrogate losses with Fisher consistency and a sharp generalization bound, validated on benchmark datasets.
Multi-label ranking, which returns multiple top-ranked labels for each instance, has a wide range of applications for visual tasks. Due to its complicated setting, prior arts have proposed various measures to evaluate model performances. However, both theoretical analysis and empirical observations show that a model might perform inconsistently on different measures. To bridge this gap, this paper proposes a novel measure named Top-K Pairwise Ranking (TKPR), and a series of analyses show that TKPR is compatible with existing ranking-based measures. In light of this, we further establish an empirical surrogate risk minimization framework for TKPR. On one hand, the proposed framework enjoys convex surrogate losses with the theoretical support of Fisher consistency. On the other hand, we establish a sharp generalization bound for the proposed framework based on a novel technique named data-dependent contraction. Finally, empirical results on benchmark datasets validate the effectiveness of the proposed framework.