S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning
This addresses a specific bottleneck in DML for visual similarity applications, offering a practical solution with incremental improvements.
The paper tackles the trade-off between embedding dimensionality and retrieval cost in Deep Metric Learning by proposing S2SD, a method that uses knowledge distillation from high-dimensional spaces to improve performance without increasing test-time cost, achieving up to 7% improvement in Recall@1 and setting a new state-of-the-art.
Deep Metric Learning (DML) provides a crucial tool for visual similarity and zero-shot applications by learning generalizing embedding spaces, although recent work in DML has shown strong performance saturation across training objectives. However, generalization capacity is known to scale with the embedding space dimensionality. Unfortunately, high dimensional embeddings also create higher retrieval cost for downstream applications. To remedy this, we propose \emph{Simultaneous Similarity-based Self-distillation (S2SD). S2SD extends DML with knowledge distillation from auxiliary, high-dimensional embedding and feature spaces to leverage complementary context during training while retaining test-time cost and with negligible changes to the training time. Experiments and ablations across different objectives and standard benchmarks show S2SD offers notable improvements of up to 7% in Recall@1, while also setting a new state-of-the-art. Code available at https://github.com/MLforHealth/S2SD.