ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
This addresses the need for efficient continuous adaptation of neural networks on resource-constrained devices, representing a novel method rather than an incremental improvement.
The paper tackles the problem of slow on-device training by proposing ElasticTrainer, a technique that allows fully elastic runtime adaptation of trainable neural network portions, achieving up to 3.5x more training speedup and 2x-3x more energy reduction compared to existing schemes without noticeable accuracy loss.
On-device training is essential for neural networks (NNs) to continuously adapt to new online data, but can be time-consuming due to the device's limited computing power. To speed up on-device training, existing schemes select trainable NN portion offline or conduct unrecoverable selection at runtime, but the evolution of trainable NN portion is constrained and cannot adapt to the current need for training. Instead, runtime adaptation of on-device training should be fully elastic, i.e., every NN substructure can be freely removed from or added to the trainable NN portion at any time in training. In this paper, we present ElasticTrainer, a new technique that enforces such elasticity to achieve the required training speedup with the minimum NN accuracy loss. Experiment results show that ElasticTrainer achieves up to 3.5x more training speedup in wall-clock time and reduces energy consumption by 2x-3x more compared to the existing schemes, without noticeable accuracy loss.