A Note on Knowledge Distillation Loss Function for Object Classification
arXiv:2109.06458v32 citations
AI Analysis
It clarifies theoretical connections for researchers in model compression, but is incremental with no new empirical results.
The paper explains the knowledge distillation loss function for object classification, linking it to logits matching and framing it as output regularization related to label smoothing and entropy-based regularization.
This research note provides a quick introduction to the knowledge distillation loss function used in object classification. In particular, we discuss its connection to a previously proposed logits matching loss function. We further treat knowledge distillation as a specific form of output regularization and demonstrate its connection to label smoothing and entropy-based regularization.