Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning
This work addresses filter pruning for neural network compression, offering a practical solution for deployment on low-spec workstations, but it is incremental as it builds on existing NAS-based pruning methods.
The paper tackles the problem of filter pruning by proposing an Ensemble Knowledge Guidance (EKG) method to search for sub-networks with smooth loss landscapes and fine-tune them efficiently, achieving 45.04% FLOPS reduction in ResNet-50 without performance degradation in 315 GPU hours.
Conventional NAS-based pruning algorithms aim to find the sub-network with the best validation performance. However, validation performance does not successfully represent test performance, i.e., potential performance. Also, although fine-tuning the pruned network to restore the performance drop is an inevitable process, few studies have handled this issue. This paper provides a novel Ensemble Knowledge Guidance (EKG) to solve both problems at once. First, we experimentally prove that the fluctuation of loss landscape can be an effective metric to evaluate the potential performance. In order to search a sub-network with the smoothest loss landscape at a low cost, we employ EKG as a search reward. EKG utilized for the following search iteration is composed of the ensemble knowledge of interim sub-networks, i.e., the by-products of the sub-network evaluation. Next, we reuse EKG to provide a gentle and informative guidance to the pruned network while fine-tuning the pruned network. Since EKG is implemented as a memory bank in both phases, it requires a negligible cost. For example, when pruning and training ResNet-50, just 315 GPU hours are required to remove around 45.04% of FLOPS without any performance degradation, which can operate even on a low-spec workstation. the implemented code is available at https://github.com/sseung0703/EKG.