Compressing image encoders via latent distillation
This addresses hardware constraints in image compression applications, offering a practical solution for resource-limited environments, though it is incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of compressing deep learning models for image compression by reducing encoder size, using a simplified knowledge distillation strategy to approximate latent spaces with less data and training. Experiments show it preserves reconstruction quality and statistical fidelity better than training lightweight encoders with original loss, making it practical for resource-limited environments.
Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to partially compress these networks by reducing the size of their encoders. Our approach uses a simplified knowledge distillation strategy to approximate the latent space of the original models with less data and shorter training, yielding lightweight encoders from heavyweight ones. We evaluate the resulting lightweight encoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity better than training lightweight encoders with the original loss, making it practical for resource-limited environments.