ClassPruning: Speed Up Image Restoration Networks by Dynamic N:M Pruning
This addresses computational inefficiency in image restoration for applications requiring real-time or resource-constrained processing, but it is incremental as it builds on existing pruning and classification methods.
The paper tackles the inefficiency of static deep neural networks in image restoration by proposing ClassPruning, a pipeline that dynamically prunes networks based on image difficulty, saving about 40% FLOPs while maintaining performance.
Image restoration tasks have achieved tremendous performance improvements with the rapid advancement of deep neural networks. However, most prevalent deep learning models perform inference statically, ignoring that different images have varying restoration difficulties and lightly degraded images can be well restored by slimmer subnetworks. To this end, we propose a new solution pipeline dubbed ClassPruning that utilizes networks with different capabilities to process images with varying restoration difficulties. In particular, we use a lightweight classifier to identify the image restoration difficulty, and then the sparse subnetworks with different capabilities can be sampled based on predicted difficulty by performing dynamic N:M fine-grained structured pruning on base restoration networks. We further propose a novel training strategy along with two additional loss terms to stabilize training and improve performance. Experiments demonstrate that ClassPruning can help existing methods save approximately 40% FLOPs while maintaining performance.