Pruning Early Exit Networks
This work addresses computational efficiency for deep learning practitioners, but it is incremental as it combines existing techniques.
The paper tackles the problem of high computational costs in deep learning models by combining pruning and early exit networks, finding that pruning the entire network at once is generally better, but both approaches perform similarly at high accuracy rates, indicating separability without loss of optimality.
Deep learning models that perform well often have high computational costs. In this paper, we combine two approaches that try to reduce the computational cost while keeping the model performance high: pruning and early exit networks. We evaluate two approaches of pruning early exit networks: (1) pruning the entire network at once, (2) pruning the base network and additional linear classifiers in an ordered fashion. Experimental results show that pruning the entire network at once is a better strategy in general. However, at high accuracy rates, the two approaches have a similar performance, which implies that the processes of pruning and early exit can be separated without loss of optimality.