Dissecting the impact of different loss functions with gradient surgery
This work addresses the challenge of optimizing metric learning for image retrieval, providing insights into loss function design that could benefit researchers and practitioners in computer vision.
The authors tackled the problem of understanding and improving pair-wise loss functions in metric learning by decomposing their gradients into components affecting anchor-positive and anchor-negative pairs, leading to a new algorithm that achieved state-of-the-art results on image retrieval datasets such as CAR, CUB, and Stanford Online Products.
Pair-wise loss is an approach to metric learning that learns a semantic embedding by optimizing a loss function that encourages images from the same semantic class to be mapped closer than images from different classes. The literature reports a large and growing set of variations of the pair-wise loss strategies. Here we decompose the gradient of these loss functions into components that relate to how they push the relative feature positions of the anchor-positive and anchor-negative pairs. This decomposition allows the unification of a large collection of current pair-wise loss functions. Additionally, explicitly constructing pair-wise gradient updates to separate out these effects gives insights into which have the biggest impact, and leads to a simple algorithm that beats the state of the art for image retrieval on the CAR, CUB and Stanford Online products datasets.