Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
This work addresses memory and computational constraints for on-device learning, which is crucial for deploying AI on resource-limited devices, though it appears incremental as an alternative to existing low-rank decomposition methods.
The paper tackles the problem of high memory and computational demands in on-device learning by proposing a shortcut approach, achieving up to 120.09x reduction in activation memory usage and up to 1.86x reduction in training FLOPs compared to vanilla training.
On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.