Compressing Model with Few Class-Imbalance Samples: An Out-of-Distribution Expedition
This addresses a common but overlooked issue in model compression for privacy-sensitive applications with limited data, though it is incremental as it builds on existing methods.
The paper tackles the problem of class imbalance in few-sample model compression, which negatively affects performance, and proposes an adaptive framework using out-of-distribution data to rebalance training, showing it mitigates accuracy degradation in experiments on multiple benchmark datasets.
In recent years, as a compromise between privacy and performance, few-sample model compression has been widely adopted to deal with limited data resulting from privacy and security concerns. However, when the number of available samples is extremely limited, class imbalance becomes a common and tricky problem. Achieving an equal number of samples across all classes is often costly and impractical in real-world applications, and previous studies on few-sample model compression have mostly ignored this significant issue. Our experiments comprehensively demonstrate that class imbalance negatively affects the overall performance of few-sample model compression methods. To address this problem, we propose a novel and adaptive framework named OOD-Enhanced Few-Sample Model Compression (OE-FSMC). This framework integrates easily accessible out-of-distribution (OOD) data into both the compression and fine-tuning processes, effectively rebalancing the training distribution. We also incorporate a joint distillation loss and a regularization term to reduce the risk of the model overfitting to the OOD data. Extensive experiments on multiple benchmark datasets show that our framework can be seamlessly incorporated into existing few-sample model compression methods, effectively mitigating the accuracy degradation caused by class imbalance.