Ada-QPacknet -- adaptive pruning with bit width reduction as an efficient continual learning method without forgetting
This work addresses the challenge of continual learning in dynamic environments for deep learning models, offering a novel compression-based approach that is incremental in nature.
The authors tackled the problem of efficient continual learning by introducing Ada-QPacknet, which combines pruning and quantization to create task-specific sub-networks, achieving similar accuracy to floating-point models while reducing model size.
Continual Learning (CL) is a process in which there is still huge gap between human and deep learning model efficiency. Recently, many CL algorithms were designed. Most of them have many problems with learning in dynamic and complex environments. In this work new architecture based approach Ada-QPacknet is described. It incorporates the pruning for extracting the sub-network for each task. The crucial aspect in architecture based CL methods is theirs capacity. In presented method the size of the model is reduced by efficient linear and nonlinear quantisation approach. The method reduces the bit-width of the weights format. The presented results shows that low bit quantisation achieves similar accuracy as floating-point sub-network on a well-know CL scenarios. To our knowledge it is the first CL strategy which incorporates both compression techniques pruning and quantisation for generating task sub-networks. The presented algorithm was tested on well-known episode combinations and compared with most popular algorithms. Results show that proposed approach outperforms most of the CL strategies in task and class incremental scenarios.