SCoTTi: Save Computation at Training Time with an adaptive framework
This addresses the problem of limited computational power on edge devices for machine learning practitioners, though it appears incremental as it builds on existing methods for resource reduction.
The paper tackles the challenge of high computational resource consumption during on-device training by proposing SCoTTi, an adaptive framework that reduces neuron updates, resulting in superior computational savings compared to state-of-the-art methods on benchmarks like ResNets and MobileNet.
On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.