Empirical investigations on WVA structural issues
This work addresses incremental improvements in methods for preventing catastrophic forgetting, relevant for researchers in continual learning.
The paper investigates structural issues in the WVA method for mitigating catastrophic forgetting in neural networks, focusing on gradient application, attenuation function selection, and hyper-parameter optimization for sequential tasks.
In this paper we want to present the results of empirical verification of some issues concerning the methods for overcoming catastrophic forgetting in neural networks. First, in the introduction, we will try to describe in detail the problem of catastrophic forgetting and methods for overcoming it for those who are not yet familiar with this topic. Then we will discuss the essence and limitations of the WVA method which we presented in previous papers. Further, we will touch upon the issues of applying the WVA method to gradients or optimization steps of weights, choosing the optimal attenuation function in this method, as well as choosing the optimal hyper-parameters of the method depending on the number of tasks in sequential training of neural networks.