Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models
This work addresses a methodological gap in continual learning research, providing a strong baseline for evaluating foundation model-based methods, which is incremental but crucial for accurate progress assessment.
The paper tackles the lack of rigorous baselines for continual learning with foundation models by introducing Pruned Adaptation Modules (PAM), a method that reduces trainable parameters by up to ~5x and total parameters by ~6x while outperforming state-of-the-art approaches in mitigating catastrophic forgetting.
The continual learning literature has rapidly shifted from traditional class incremental learning (CIL) techniques to foundation model (FM)-based CIL methods without a clear understanding of how these newer approaches compare to strong, lightweight convolutional baselines. This abrupt transition has created a substantial methodological gap, making it difficult to assess whether recent FM-based CIL progress reflects genuine advances or merely the absence of rigorous baselines. To address this gap, we introduce Pruned Adaptation Modules (PAM), a simple yet effective method that freezes the vast majority of the pre-trained ResNet while enabling scalable continual adaptation through sparse task-specific layers. PAM yields up to a ~5x reduction in trainable parameters and a ~6x reduction in total parameters, significantly reducing the cost of continual updates. Across diverse benchmarks, PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches. Our findings position PAM as a strong and transparent baseline that helps bridge the gap between traditional and FM-based CIL, guiding future research for a more accurate assessment of true progress in continual adaptation. The code can be found at: https://github.com/ElifCerenGokYildirim/PAM.