Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks
This addresses the challenge of deploying continual learning on resource-constrained edge devices, offering a scalable solution, though it appears incremental as it builds on existing sparsity and competition principles.
The paper tackles the problem of continual learning on edge devices by introducing a method that uses stochastic local competition among subnetworks to achieve sparsity, reducing memory footprint and computational demand, with results showing sparse task-specific representations that facilitate efficient training and inference.
Continual learning on edge devices poses unique challenges due to stringent resource constraints. This paper introduces a novel method that leverages stochastic competition principles to promote sparsity, significantly reducing deep network memory footprint and computational demand. Specifically, we propose deep networks that comprise blocks of units that compete locally to win the representation of each arising new task; competition takes place in a stochastic manner. This type of network organization results in sparse task-specific representations from each network layer; the sparsity pattern is obtained during training and is different among tasks. Crucially, our method sparsifies both the weights and the weight gradients, thus facilitating training on edge devices. This is performed on the grounds of winning probability for each unit in a block. During inference, the network retains only the winning unit and zeroes-out all weights pertaining to non-winning units for the task at hand. Thus, our approach is specifically tailored for deployment on edge devices, providing an efficient and scalable solution for continual learning in resource-limited environments.