Ranking vs. Assignment: The Metric Mismatch in Multi-View Object Association
For researchers in multi-view object association, this work highlights a critical evaluation flaw and provides a stress test to detect it, but the solution is incremental.
The paper identifies a mismatch between ranking metrics (AP, FPR-95) and the actual assignment objective in multi-view object association, showing that optimizing ranking metrics does not guarantee correct assignments. A Sinkhorn-based normalization post-processing method is proposed that improves ranking metrics without improving assignment-level metrics, confirming the mismatch.
Multi-view object association is an important computer vision problem that underlies many multi-camera perception tasks. While this task is naturally formulated as a constrained one-to-one matching problem, recent works heavily rely on pairwise ranking metrics like AP and FPR-95 for model evaluation. We highlight a fundamental mismatch between these metrics and the actual assignment objective. Theoretically, we show that AP and FPR-95 can be imperfect even when the assignment is already correct, and that Sinkhorn-based normalization can make them perfect. Conversely, optimal pairwise ranking can still lead to incorrect assignments. We validate this mismatch in practice by using our Sinkhorn-based normalization as a controlled post-processing stress test. We show that optimizing just a few post-processing parameters significantly boosts AP and FPR-95 without corresponding improvements in assignment-level metrics such as ACC and IPAA.