Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning
This work addresses the problem of auto pruning for deep CNNs, offering an incremental improvement in usability and efficiency for model compression tasks.
The paper tackles the challenge of efficiently finding optimal pruning policies for deep convolutional neural networks (CNNs) by proposing a clustering and rollback method to reduce dimensionality in Bayesian optimization, resulting in significantly improved convergence rates without increased running time.
Pruning is an effective technique for convolutional neural networks (CNNs) model compression, but it is difficult to find the optimal pruning policy due to the large design space. To improve the usability of pruning, many auto pruning methods have been developed. Recently, Bayesian optimization (BO) has been considered to be a competitive algorithm for auto pruning due to its solid theoretical foundation and high sampling efficiency. However, BO suffers from the curse of dimensionality. The performance of BO deteriorates when pruning deep CNNs, since the dimension of the design spaces increase. We propose a novel clustering algorithm that reduces the dimension of the design space to speed up the searching process. Subsequently, a rollback algorithm is proposed to recover the high-dimensional design space so that higher pruning accuracy can be obtained. We validate our proposed method on ResNet, MobileNetV1, and MobileNetV2 models. Experiments show that the proposed method significantly improves the convergence rate of BO when pruning deep CNNs with no increase in running time. The source code is available at https://github.com/fanhanwei/BOCR.