A Conformal Predictive Measure for Assessing Catastrophic Forgetting
This provides a practical tool for researchers and practitioners in machine learning to monitor forgetting in dynamic learning environments, though it is incremental as it builds on existing conformal prediction methods.
The paper tackles the problem of assessing catastrophic forgetting in continual learning by introducing a new conformal prediction-based metric called CPCF, which shows a strong correlation with accuracy on previous tasks in experiments across four benchmark datasets.
This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF), to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model's confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments.