Optimal Task Order for Continual Learning of Multiple Tasks
This work addresses a key challenge in continual learning for neural networks, offering a generalizable framework that is incremental but provides specific gains for task-incremental scenarios.
The paper tackled the problem of how task order influences continual learning performance by deriving analytical principles for optimal ordering, showing that arranging tasks from least to most representative and keeping adjacent tasks dissimilar improves results, with validation on synthetic and real datasets like CIFAR-10 and CIFAR-100.
Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it. Leveraging a linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), demonstrating consistent performance improvements in both multilayer perceptrons and convolutional neural networks. Our work thus presents a generalizable framework for task-order optimization in task-incremental continual learning.