Comprehensive Survey of Model Compression and Speed up for Vision Transformers
It addresses the problem of deploying ViTs in resource-constrained environments, such as edge computing, but is incremental as it surveys and compares existing techniques without introducing new methods.
This study tackled the high computational and memory demands of Vision Transformers (ViT) by evaluating four model compression techniques—quantization, low-rank approximation, knowledge distillation, and pruning—and their combinations, demonstrating that these methods enable a balanced trade-off between accuracy and efficiency for deployment in edge computing devices.
Vision Transformers (ViT) have marked a paradigm shift in computer vision, outperforming state-of-the-art models across diverse tasks. However, their practical deployment is hampered by high computational and memory demands. This study addresses the challenge by evaluating four primary model compression techniques: quantization, low-rank approximation, knowledge distillation, and pruning. We methodically analyze and compare the efficacy of these techniques and their combinations in optimizing ViTs for resource-constrained environments. Our comprehensive experimental evaluation demonstrates that these methods facilitate a balanced compromise between model accuracy and computational efficiency, paving the way for wider application in edge computing devices.