Bringing Giant Neural Networks Down to Earth with Unlabeled Data
This addresses the challenge of deploying large models on edge devices where data privacy or availability limits retraining, though it is an incremental improvement on existing compression methods.
The paper tackles the problem of compressing giant neural networks when only a fraction of labeled training data is available, by using unlabeled data to mimic classification characteristics and aligning distributions with an adversarial loss, resulting in significantly improved performance for compressed networks.
Compressing giant neural networks has gained much attention for their extensive applications on edge devices such as cellphones. During the compressing process, one of the most important procedures is to retrain the pre-trained models using the original training dataset. However, due to the consideration of security, privacy or commercial profits, in practice, only a fraction of sample training data are made available, which makes the retraining infeasible. To solve this issue, this paper proposes to resort to unlabeled data in hand that can be cheaper to acquire. Specifically, we exploit the unlabeled data to mimic the classification characteristics of giant networks, so that the original capacity can be preserved nicely. Nevertheless, there exists a dataset bias between the labeled and unlabeled data, which may disturb the training and degrade the performance. We thus fix this bias by an adversarial loss to make an alignment on the distributions of their low-level feature representations. We further provide theoretical discussions about how the unlabeled data help compressed networks to generalize better. Experimental results demonstrate that the unlabeled data can significantly improve the performance of the compressed networks.