Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification
This work addresses the problem of computational efficiency and privacy in image classification for AI practitioners, but it is incremental as it builds on existing low-rank adaptation methods.
The paper tackles efficient fine-tuning of vision transformers for privacy-preserving image classification by proposing a low-rank adaptation method that freezes pre-trained weights and injects trainable rank decomposition matrices, achieving nearly the same accuracy as full fine-tuning while reducing trainable parameters.
We propose a low-rank adaptation method for training privacy-preserving vision transformer (ViT) models that efficiently freezes pre-trained ViT model weights. In the proposed method, trainable rank decomposition matrices are injected into each layer of the ViT architecture, and moreover, the patch embedding layer is not frozen, unlike in the case of the conventional low-rank adaptation methods. The proposed method allows us not only to reduce the number of trainable parameters but to also maintain almost the same accuracy as that of full-time tuning.