FCL-ViT: Task-Aware Attention Tuning for Continual Learning
This addresses the challenge of adapting prior knowledge to new tasks without forgetting old ones in continual learning, offering a domain-specific improvement.
The paper tackles the problem of continual learning in deep neural networks by introducing FCL-ViT, a novel Vision Transformer that uses a feedback mechanism to generate dynamic attention features tailored to each task, achieving state-of-the-art performance with a small number of trainable parameters.
Continual Learning (CL) involves adapting the prior Deep Neural Network (DNN) knowledge to new tasks, without forgetting the old ones. However, modern CL techniques focus on provisioning memory capabilities to existing DNN models rather than designing new ones that are able to adapt according to the task at hand. This paper presents the novel Feedback Continual Learning Vision Transformer (FCL-ViT) that uses a feedback mechanism to generate real-time dynamic attention features tailored to the current task. The FCL-ViT operates in two Phases. In phase 1, the generic image features are produced and determine where the Transformer should attend on the current image. In phase 2, task-specific image features are generated that leverage dynamic attention. To this end, Tunable self-Attention Blocks (TABs) and Task Specific Blocks (TSBs) are introduced that operate in both phases and are responsible for tuning the TABs attention, respectively. The FCL-ViT surpasses state-of-the-art performance on Continual Learning compared to benchmark methods, while retaining a small number of trainable DNN parameters.