Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning
This work addresses stabilization challenges in continual learning methods for deep neural networks, but it is incremental as it builds on existing EWC techniques.
The paper tackles the practical application of Elastic Weight Consolidation (EWC) for continual learning in neural networks, comparing methodologies like MAS, SI, and Fisher-based importance, and proposes a stabilization approach that performs no worse than original EWC while avoiding issues like gradient explosion.
This paper is devoted to the features of the practical application of the Elastic Weight Consolidation (EWC) method for continual learning of neural networks on several training sets. We will more rigorously compare the well-known methodologies for calculating the importance of weights used in the EWC method. These are the Memory Aware Synapses (MAS), Synaptic Intelligence (SI) methodologies and the calculation of the importance of weights based on the Fisher information matrix from the original paper on EWC. We will consider these methodologies as applied to deep neural networks with fully connected and convolutional layers, find the optimal hyperparameters for each of the methodologies, and compare the results of continual neural network learning using these hyperparameters. Next, we will point out the problems that arise when applying the EWC method to deep neural networks with convolutional layers and self-attention layers, such as the "gradient explosion" and the loss of meaningful information in the gradient when using the constraint of its norm (gradient clipping). Then, we will propose a stabilization approach for the EWC method that helps to solve these problems, evaluate it in comparison with the original methodology and show that the proposed stabilization approach performs on the task of maintaining skills during continual learning no worse than the original EWC, but does not have its disadvantages. In conclusion, we present an interesting fact about the use of various types of weight importance in the problem of neural network pruning.