Monte Carlo Functional Regularisation for Continual Learning
This addresses computational bottlenecks in continual learning for neural networks, though it appears incremental as it builds on existing functional regularization approaches.
The paper tackles the computational cost and approximation error problems in functional regularization methods for continual learning by proposing MCFRCL, which uses Monte Carlo sampling and statistical distributions to approximate model predictions, achieving improved accuracy and efficiency on MNIST and CIFAR benchmarks.
Continual learning (CL) is crucial for the adaptation of neural network models to new environments. Although outperforming weight-space regularisation approaches, the functional regularisation-based CL methods suffer from high computational costs and large linear approximation errors. In this work, we present a new functional regularisation CL framework, called MCFRCL, which approximates model prediction distributions by Monte Carlo (MC) sampling. Moreover, three continuous distributions are leveraged to capture the statistical characteristics of the MC samples via moment-based methods. Additionally, both the Wasserstein distance and the Kullback-Leibler (KL) distance are employed to construct the regularisation function. The proposed MCFRCL is evaluated against multiple benchmark methods on the MNIST and CIFAR datasets, with simulation results highlighting its effectiveness in both prediction accuracy and training efficiency.