Compressing Models with Few Samples: Mimicking then Replacing
This addresses the challenge of model compression with limited data, which is crucial for deploying AI in resource-constrained environments, though it appears incremental as it builds on existing few-sample compression techniques.
The paper tackles the problem of compressing large models into smaller ones using only a few samples, which typically leads to overfitting. The proposed Mimicking then Replacing (MiR) framework outperforms previous methods by large margins, offering a simpler and more effective approach.
Few-sample compression aims to compress a big redundant model into a small compact one with only few samples. If we fine-tune models with these limited few samples directly, models will be vulnerable to overfit and learn almost nothing. Hence, previous methods optimize the compressed model layer-by-layer and try to make every layer have the same outputs as the corresponding layer in the teacher model, which is cumbersome. In this paper, we propose a new framework named Mimicking then Replacing (MiR) for few-sample compression, which firstly urges the pruned model to output the same features as the teacher's in the penultimate layer, and then replaces teacher's layers before penultimate with a well-tuned compact one. Unlike previous layer-wise reconstruction methods, our MiR optimizes the entire network holistically, which is not only simple and effective, but also unsupervised and general. MiR outperforms previous methods with large margins. Codes will be available soon.