Efficient Resource-Constrained Training of Vision Transformers via Subspace Optimization
This work addresses energy consumption and data privacy issues for edge devices by enabling efficient training of transformer models, though it is incremental as it applies subspace optimization to an existing model type.
The paper tackles the challenge of on-device training for vision transformers under resource constraints by introducing Weight-Activation Subspace Iteration (WASI), which reduces memory usage by up to 62× and computational cost by up to 2× while maintaining comparable accuracy.
As AI increasingly shapes daily life, energy consumption and data privacy have become pressing concerns. On-device learning trains models directly on edge devices, cutting energy consumption and safeguarding data privacy. However, the expanding scale of modern neural networks creates a major obstacle for on-device training. Although prior work has concentrated on compact convolutional architectures, we instead apply subspace-based training to transformer models. Motivated by the idea that a model's essential information lies in a fixed subspace, we introduce Weight-Activation Subspace Iteration (WASI), a method that mitigates the memory bottleneck of backpropagation and boosts inference efficiency in transformer models by restricting training to this subspace. Our results demonstrate that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to $62\times$ and computational cost (FLOPs) by up to $2\times$. On a Raspberry Pi 5, WASI achieves roughly $1.5\times$ faster training and inference than vanilla training.