Vision Transformer Compression with Structured Pruning and Low Rank Approximation
This work addresses the need for efficient vision transformer models for on-device deployment, presenting an incremental improvement over existing compression methods.
The paper tackles the problem of compressing vision transformers for on-device deployment by proposing a hybrid compression approach combining low-rank approximation and structured pruning, achieving 50% compression with a 14% relative increase in classification error compared to 44% compression with a 20% error increase using pruning alone.
Transformer architecture has gained popularity due to its ability to scale with large dataset. Consequently, there is a need to reduce the model size and latency, especially for on-device deployment. We focus on vision transformer proposed for image recognition task (Dosovitskiy et al., 2021), and explore the application of different compression techniques such as low rank approximation and pruning for this purpose. Specifically, we investigate a structured pruning method proposed recently in Zhu et al. (2021) and find that mostly feedforward blocks are pruned with this approach, that too, with severe degradation in accuracy. We propose a hybrid compression approach to mitigate this where we compress the attention blocks using low rank approximation and use the previously mentioned pruning with a lower rate for feedforward blocks in each transformer layer. Our technique results in 50% compression with 14% relative increase in classification error whereas we obtain 44% compression with 20% relative increase in error when only pruning is applied. We propose further enhancements to bridge the accuracy gap but leave it as a future work.