Hard ASH: Sparsity and the right optimizer make a continual learner
This work addresses the problem of catastrophic forgetting in neural networks for continual learning, offering an incremental improvement for researchers in this domain.
The paper tackled catastrophic forgetting in class incremental learning by introducing Hard Adaptive SwisH (Hard ASH), a sparse activation function variant, combined with an adaptive learning rate optimizer, achieving competitive performance with established regularization techniques on the Split-MNIST task.
In class incremental learning, neural networks typically suffer from catastrophic forgetting. We show that an MLP featuring a sparse activation function and an adaptive learning rate optimizer can compete with established regularization techniques in the Split-MNIST task. We highlight the effectiveness of the Adaptive SwisH (ASH) activation function in this context and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further enhance the learning retention.