A Simple Baseline for Stable and Plastic Neural Networks
This provides a practical solution for real-world continual learning applications, though it is incremental as it builds on existing approaches.
The paper tackles the problem of balancing plasticity and stability in continual learning for computer vision, introducing RDBP, which matches or exceeds state-of-the-art methods on the Continual ImageNet benchmark while reducing computational cost.
Continual learning in computer vision requires that models adapt to a continuous stream of tasks without forgetting prior knowledge, yet existing approaches often tip the balance heavily toward either plasticity or stability. We introduce RDBP, a simple, low-overhead baseline that unites two complementary mechanisms: ReLUDown, a lightweight activation modification that preserves feature sensitivity while preventing neuron dormancy, and Decreasing Backpropagation, a biologically inspired gradient-scheduling scheme that progressively shields early layers from catastrophic updates. Evaluated on the Continual ImageNet benchmark, RDBP matches or exceeds the plasticity and stability of state-of-the-art methods while reducing computational cost. RDBP thus provides both a practical solution for real-world continual learning and a clear benchmark against which future continual learning strategies can be measured.