Activation Function Design Sustains Plasticity in Continual Learning
This addresses the challenge of sustaining model adaptability in continual learning for AI systems, offering a lightweight, domain-general solution without extra capacity, though it is incremental as it builds on existing activation function analysis.
The paper tackled the problem of plasticity loss in continual learning by designing new activation functions, showing that Smooth-Leaky and Randomized Smooth-Leaky nonlinearities mitigate this issue, achieving up to 15% higher accuracy in class-incremental benchmarks and 20% better performance in non-stationary reinforcement learning environments.
In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differences often shrink once model size and optimization are tuned. In continual learning, however, the picture is different: beyond catastrophic forgetting, models can progressively lose the ability to adapt (referred to as loss of plasticity) and the role of the non-linearity in this failure mode remains underexplored. We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss. Building on a property-level analysis of negative-branch shape and saturation behavior, we introduce two drop-in nonlinearities (Smooth-Leaky and Randomized Smooth-Leaky) and evaluate them in two complementary settings: (i) supervised class-incremental benchmarks and (ii) reinforcement learning with non-stationary MuJoCo environments designed to induce controlled distribution and dynamics shifts. We also provide a simple stress protocol and diagnostics that link the shape of the activation to the adaptation under change. The takeaway is straightforward: thoughtful activation design offers a lightweight, domain-general way to sustain plasticity in continual learning without extra capacity or task-specific tuning.