Mathematical Justification of Hard Negative Mining via Isometric Approximation Theorem
This work addresses a theoretical gap for researchers in metric learning, offering a mathematical basis for an empirically successful technique, though it is incremental as it builds on existing mining strategies.
The paper tackles the lack of theoretical justification for hard negative mining in deep metric learning by using isometric approximation theory to show an equivalence between Triplet Loss with hard negative mining and an optimization problem minimizing a Hausdorff-like distance, providing a foundation for future methods to avoid network collapse.
In deep metric learning, the Triplet Loss has emerged as a popular method to learn many computer vision and natural language processing tasks such as facial recognition, object detection, and visual-semantic embeddings. One issue that plagues the Triplet Loss is network collapse, an undesirable phenomenon where the network projects the embeddings of all data onto a single point. Researchers predominately solve this problem by using triplet mining strategies. While hard negative mining is the most effective of these strategies, existing formulations lack strong theoretical justification for their empirical success. In this paper, we utilize the mathematical theory of isometric approximation to show an equivalence between the Triplet Loss sampled by hard negative mining and an optimization problem that minimizes a Hausdorff-like distance between the neural network and its ideal counterpart function. This provides the theoretical justifications for hard negative mining's empirical efficacy. In addition, our novel application of the isometric approximation theorem provides the groundwork for future forms of hard negative mining that avoid network collapse. Our theory can also be extended to analyze other Euclidean space-based metric learning methods like Ladder Loss or Contrastive Learning.