Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation
This work addresses efficiency issues in image-to-image translation for applications requiring faster deployment, though it is incremental as it adapts existing compression techniques to a new task.
The paper tackles the problem of large model size and slow inference in GAN-based image-to-image translation by proposing a knowledge distillation method that preserves semantic relations, achieving strong qualitative and quantitative results across multiple datasets and model pairs.
Generative adversarial networks (GANs) have shown significant potential in modeling high dimensional distributions of image data, especially on image-to-image translation tasks. However, due to the complexity of these tasks, state-of-the-art models often contain a tremendous amount of parameters, which results in large model size and long inference time. In this work, we propose a novel method to address this problem by applying knowledge distillation together with distillation of a semantic relation preserving matrix. This matrix, derived from the teacher's feature encoding, helps the student model learn better semantic relations. In contrast to existing compression methods designed for classification tasks, our proposed method adapts well to the image-to-image translation task on GANs. Experiments conducted on 5 different datasets and 3 different pairs of teacher and student models provide strong evidence that our methods achieve impressive results both qualitatively and quantitatively.