KASAM: Spline Additive Models for Function Approximation
This addresses catastrophic forgetting in continual learning for AI systems, but it is incremental as it builds on existing models and training techniques.
The paper tackles catastrophic forgetting in neural networks by proposing a Spline Additive Model (SAM) with intrinsic memory retention and extending it to a universal approximator called KASAM using the Kolmogorov-Arnold theorem, finding that SAM shows robust but imperfect retention while KASAM with pseudo-rehearsal achieves superior performance in regression tasks.
Neural networks have been criticised for their inability to perform continual learning due to catastrophic forgetting and rapid unlearning of a past concept when a new concept is introduced. Catastrophic forgetting can be alleviated by specifically designed models and training techniques. This paper outlines a novel Spline Additive Model (SAM). SAM exhibits intrinsic memory retention with sufficient expressive power for many practical tasks, but is not a universal function approximator. SAM is extended with the Kolmogorov-Arnold representation theorem to a novel universal function approximator, called the Kolmogorov-Arnold Spline Additive Model - KASAM. The memory retention, expressive power and limitations of SAM and KASAM are illustrated analytically and empirically. SAM exhibited robust but imperfect memory retention, with small regions of overlapping interference in sequential learning tasks. KASAM exhibited greater susceptibility to catastrophic forgetting. KASAM in combination with pseudo-rehearsal training techniques exhibited superior performance in regression tasks and memory retention.