Optimizing Rank-based Metrics with Blackbox Differentiation
This addresses a long-standing problem in computer vision for researchers and practitioners by enabling direct optimization of widely used evaluation metrics, though it is incremental as it builds on existing methods.
The paper tackles the challenge of directly optimizing non-differentiable rank-based metrics like recall and Average Precision by introducing an efficient blackbox differentiation method, achieving competitive state-of-the-art results in image retrieval and consistent improvements in object detection tasks.
Rank-based metrics are some of the most widely used criteria for performance evaluation of computer vision models. Despite years of effort, direct optimization for these metrics remains a challenge due to their non-differentiable and non-decomposable nature. We present an efficient, theoretically sound, and general method for differentiating rank-based metrics with mini-batch gradient descent. In addition, we address optimization instability and sparsity of the supervision signal that both arise from using rank-based metrics as optimization targets. Resulting losses based on recall and Average Precision are applied to image retrieval and object detection tasks. We obtain performance that is competitive with state-of-the-art on standard image retrieval datasets and consistently improve performance of near state-of-the-art object detectors. The code is available at https://github.com/martius-lab/blackbox-backprop