Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting
This work provides insights into why diverse weight importance methods are effective in EWC, which is incremental for researchers in continual learning.
The study investigated the correlation between different methods for calculating neural network weight importances used in Elastic Weight Consolidation (EWC) to address catastrophic forgetting, finding that some methods strongly correlate while others vary significantly across network layers.
Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?