Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling
This work addresses a practical issue for researchers and practitioners in continual learning by enabling fairer comparisons and more efficient resource usage, though it is incremental as it builds on existing CL methods with specific optimizations.
The paper tackled the problem of unfair comparisons in online continual learning due to varying computational and storage budgets, proposing to use FLOPs and memory size as metrics and introducing adaptive layer freezing and frequency-based sampling to improve performance within a fixed total budget. The result showed that the proposed approach outperformed state-of-the-art methods on datasets like CIFAR-10/100, CLEAR-10/100, and ImageNet-1K.
The majority of online continual learning (CL) advocates single-epoch training and imposes restrictions on the size of replay memory. However, single-epoch training would incur a different amount of computations per CL algorithm, and the additional storage cost to store logit or model in addition to replay memory is largely ignored in calculating the storage budget. Arguing different computational and storage budgets hinder fair comparison among CL algorithms in practice, we propose to use floating point operations (FLOPs) and total memory size in Byte as a metric for computational and memory budgets, respectively, to compare and develop CL algorithms in the same 'total resource budget.' To improve a CL method in a limited total budget, we propose adaptive layer freezing that does not update the layers for less informative batches to reduce computational costs with a negligible loss of accuracy. In addition, we propose a memory retrieval method that allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations. Empirical validations on the CIFAR-10/100, CLEAR-10/100, and ImageNet-1K datasets demonstrate that the proposed approach outperforms the state-of-the-art methods within the same total budget