The impact of model size on catastrophic forgetting in Online Continual Learning
This addresses model scalability issues for researchers and practitioners in Continual Learning, but it is incremental as it builds on existing knowledge about catastrophic forgetting.
The study investigated how model size affects catastrophic forgetting in Online Continual Learning, finding that larger models often perform worse, challenging the assumption that bigger models reduce forgetting.
This study investigates the impact of model size on Online Continual Learning performance, with a focus on catastrophic forgetting. Employing ResNet architectures of varying sizes, the research examines how network depth and width affect model performance in class-incremental learning using the SplitCIFAR-10 dataset. Key findings reveal that larger models do not guarantee better Continual Learning performance; in fact, they often struggle more in adapting to new tasks, particularly in online settings. These results challenge the notion that larger models inherently mitigate catastrophic forgetting, highlighting the nuanced relationship between model size and Continual Learning efficacy. This study contributes to a deeper understanding of model scalability and its practical implications in Continual Learning scenarios.