Learning Equi-angular Representations for Online Continual Learning
This addresses the challenge of efficient model updates in online continual learning for AI systems handling streaming data, representing an incremental improvement over existing methods.
The paper tackles the underfitting problem in online continual learning caused by single-epoch training by inducing neural collapse to form a simplex equiangular tight frame structure in the representation space, resulting in outperforming state-of-the-art methods by a noticeable margin across datasets like CIFAR-10/100 and ImageNet-1K.
Online continual learning suffers from an underfitted solution due to insufficient training for prompt model update (e.g., single-epoch training). To address the challenge, we propose an efficient online continual learning method using the neural collapse phenomenon. In particular, we induce neural collapse to form a simplex equiangular tight frame (ETF) structure in the representation space so that the continuously learned model with a single epoch can better fit to the streamed data by proposing preparatory data training and residual correction in the representation space. With an extensive set of empirical validations using CIFAR-10/100, TinyImageNet, ImageNet-200, and ImageNet-1K, we show that our proposed method outperforms state-of-the-art methods by a noticeable margin in various online continual learning scenarios such as disjoint and Gaussian scheduled continuous (i.e., boundary-free) data setups.