Conditional updates of neural network weights for increased out of training performance
This addresses the challenge of adapting neural networks to new or shifted data patterns, particularly in climate sciences, but is incremental as it builds on existing retraining and regression techniques.
The study tackled the problem of neural network performance degradation when training and application data differ, such as in out-of-distribution scenarios, by proposing a method to extrapolate network weights based on predictors derived from training subsets, achieving successful extrapolations in temporal, spatial, and cross-domain climate science use cases.
This study proposes a method to enhance neural network performance when training data and application data are not very similar, e.g., out of distribution problems, as well as pattern and regime shifts. The method consists of three main steps: 1) Retrain the neural network towards reasonable subsets of the training data set and note down the resulting weight anomalies. 2) Choose reasonable predictors and derive a regression between the predictors and the weight anomalies. 3) Extrapolate the weights, and thereby the neural network, to the application data. We show and discuss this method in three use cases from the climate sciences, which include successful temporal, spatial and cross-domain extrapolations of neural networks.