Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval
This work addresses the problem of large-scale image retrieval for researchers and practitioners by providing a scalable method to directly optimize ranking metrics, though it is incremental as it builds on existing deep metric learning approaches.
The paper tackles the challenge of optimizing non-differentiable ranking metrics like Average Precision (AP) by introducing Smooth-AP, a smoothed approximation that enables end-to-end training of deep networks, resulting in improved state-of-the-art performance on multiple retrieval benchmarks, including large-scale datasets like INaturalist and VGGFace2.
Optimising a ranking-based metric, such as Average Precision (AP), is notoriously challenging due to the fact that it is non-differentiable, and hence cannot be optimised directly using gradient-descent methods. To this end, we introduce an objective that optimises instead a smoothed approximation of AP, coined Smooth-AP. Smooth-AP is a plug-and-play objective function that allows for end-to-end training of deep networks with a simple and elegant implementation. We also present an analysis for why directly optimising the ranking based metric of AP offers benefits over other deep metric learning losses. We apply Smooth-AP to standard retrieval benchmarks: Stanford Online products and VehicleID, and also evaluate on larger-scale datasets: INaturalist for fine-grained category retrieval, and VGGFace2 and IJB-C for face retrieval. In all cases, we improve the performance over the state-of-the-art, especially for larger-scale datasets, thus demonstrating the effectiveness and scalability of Smooth-AP to real-world scenarios.