Preserving Angles Improves Feature Distillation
This work addresses the challenge of effectively transferring properties from large foundation models to smaller, lightweight models for computer vision applications, offering an incremental improvement over existing feature distillation methods.
The paper tackles the problem of feature distillation failing to preserve teacher model properties like robustness and out-of-distribution detection by introducing CosPress, a technique that compresses the teacher's latent space while preserving cosine similarities between embeddings, resulting in more accurate and generalizable student models across datasets such as ImageNet.
Knowledge distillation methods compress models by training a student network using the classification outputs of a high quality teacher model, but can fail to effectively transfer the properties of computer vision foundation models from the teacher to the student. While it has been recently shown that feature distillation$\unicode{x2013}$where a teacher model's output features are replicated instead$\unicode{x2013}$can reproduce performance for foundation models across numerous downstream tasks, they fall short in matching critical properties such as robustness and out-of-distribution (OOD) detection performance. This paper overcomes this shortcoming by introducing Cosine-similarity Preserving Compression (CosPress), a feature distillation technique that learns a mapping to compress the latent space of the teacher model into the smaller latent space of the student, by preserving the cosine similarities between image embeddings. This enables direct optimisation of the student network and produces a more faithful reproduction of the teacher's properties. It is shown that distillation with CosPress on a variety of datasets, including ImageNet, produces more accurate models with greater performance on generalisability, robustness and OOD detection benchmarks, and that this technique provides a competitive pathway for training highly performant lightweight models on small datasets. Code is available at github.com/emannix/cospress.