CNN Mixture-of-Depths
This addresses computational bottlenecks for CNN users by providing a method that improves speed and efficiency without requiring complex hardware changes, though it appears incremental as it builds on existing CNN architectures.
The paper tackles the problem of computational inefficiency in CNNs by introducing Mixture-of-Depths (MoD), which selectively processes relevant channels to enhance efficiency, resulting in performance gains such as ResNet86-MoD exceeding ResNet50 by 0.45% on ImageNet with a 6% speedup on CPU.
We introduce Mixture-of-Depths (MoD) for Convolutional Neural Networks (CNNs), a novel approach that enhances the computational efficiency of CNNs by selectively processing channels based on their relevance to the current prediction. This method optimizes computational resources by dynamically selecting key channels in feature maps for focused processing within the convolutional blocks (Conv-Blocks), while skipping less relevant channels. Unlike conditional computation methods that require dynamic computation graphs, CNN MoD uses a static computation graph with fixed tensor sizes which improve hardware efficiency. It speeds up the training and inference processes without the need for customized CUDA kernels, unique loss functions, or finetuning. CNN MoD either matches the performance of traditional CNNs with reduced inference times, GMACs, and parameters, or exceeds their performance while maintaining similar inference times, GMACs, and parameters. For example, on ImageNet, ResNet86-MoD exceeds the performance of the standard ResNet50 by 0.45% with a 6% speedup on CPU and 5% on GPU. Moreover, ResNet75-MoD achieves the same performance as ResNet50 with a 25% speedup on CPU and 15% on GPU.