Adapting Models to Signal Degradation using Distillation
This addresses the challenge of adapting models to degraded signals (e.g., low-resolution images) for applications in fine-grained recognition, though it is incremental as it builds on existing distillation methods.
The paper tackles the problem of domain adaptation when aligned data is unavailable by using computer graphics to synthetically generate aligned data for distillation, improving recognition performance on low-quality data like low-resolution images and beating strong baselines.
Model compression and knowledge distillation have been successfully applied for cross-architecture and cross-domain transfer learning. However, a key requirement is that training examples are in correspondence across the domains. We show that in many scenarios of practical importance such aligned data can be synthetically generated using computer graphics pipelines allowing domain adaptation through distillation. We apply this technique to learn models for recognizing low-resolution images using labeled high-resolution images, non-localized objects using labeled localized objects, line-drawings using labeled color images, etc. Experiments on various fine-grained recognition datasets demonstrate that the technique improves recognition performance on the low-quality data and beats strong baselines for domain adaptation. Finally, we present insights into workings of the technique through visualizations and relating it to existing literature.