Continual Learning: Forget-free Winning Subnetworks for Video Representations
This work addresses the challenge of efficient and effective continual learning for video data, which is incremental in nature as it builds on the Lottery Ticket Hypothesis and adapts methods to specific learning scenarios.
The paper tackles the problem of continual learning in video representations by proposing a forget-free winning subnetwork approach, which leverages pre-existing weights and Fourier Subneural Operators to improve task performance across various scenarios, achieving significant enhancements in higher-layer performance for TIL and FSCIL and lower-layer performance for VIL.
Inspired by the Lottery Ticket Hypothesis (LTH), which highlights the existence of efficient subnetworks within larger, dense networks, a high-performing Winning Subnetwork (WSN) in terms of task performance under appropriate sparsity conditions is considered for various continual learning tasks. It leverages pre-existing weights from dense networks to achieve efficient learning in Task Incremental Learning (TIL) and Task-agnostic Incremental Learning (TaIL) scenarios. In Few-Shot Class Incremental Learning (FSCIL), a variation of WSN referred to as the Soft subnetwork (SoftNet) is designed to prevent overfitting when the data samples are scarce. Furthermore, the sparse reuse of WSN weights is considered for Video Incremental Learning (VIL). The use of Fourier Subneural Operator (FSO) within WSN is considered. It enables compact encoding of videos and identifies reusable subnetworks across varying bandwidths. We have integrated FSO into different architectural frameworks for continual learning, including VIL, TIL, and FSCIL. Our comprehensive experiments demonstrate FSO's effectiveness, significantly improving task performance at various convolutional representational levels. Specifically, FSO enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL.