Modify Training Directions in Function Space to Reduce Generalization Error
This work provides a theoretical framework for improving generalization in neural networks, which is incremental as it builds on existing methods to explain and enhance generalization results.
The authors tackled the problem of reducing generalization error in neural networks by modifying training directions in function space, using theoretical analyses based on eigendecompositions of the neural tangent kernel and Fisher information matrix, and demonstrated a reduction in total generalization error with numerical examples on synthetic data.
We propose theoretical analyses of a modified natural gradient descent method in the neural network function space based on the eigendecompositions of neural tangent kernel and Fisher information matrix. We firstly present analytical expression for the function learned by this modified natural gradient under the assumptions of Gaussian distribution and infinite width limit. Thus, we explicitly derive the generalization error of the learned neural network function using theoretical methods from eigendecomposition and statistics theory. By decomposing of the total generalization error attributed to different eigenspace of the kernel in function space, we propose a criterion for balancing the errors stemming from training set and the distribution discrepancy between the training set and the true data. Through this approach, we establish that modifying the training direction of the neural network in function space leads to a reduction in the total generalization error. Furthermore, We demonstrate that this theoretical framework is capable to explain many existing results of generalization enhancing methods. These theoretical results are also illustrated by numerical examples on synthetic data.