Observation Noise and Initialization in Wide Neural Networks
This work improves the theoretical understanding and practical application of wide neural networks for researchers in machine learning, though it is incremental by refining existing NTK-GP formulations.
The paper tackles the limitations of the Neural Tangent Kernel Gaussian Process (NTK-GP) equivalence by addressing observation noise and arbitrary prior means, showing that gradient descent in a wide neural network can correspond to a well-specified NTK-GP with noise and flexible priors, validated empirically across different noise levels and architectures.
Performing gradient descent in a wide neural network is equivalent to computing the posterior mean of a Gaussian Process with the Neural Tangent Kernel (NTK-GP), for a specific choice of prior mean and with zero observation noise. However, existing formulations of this result have two limitations: i) the resultant NTK-GP assumes no noise in the observed target variables, which can result in suboptimal predictions with noisy data; ii) it is unclear how to extend the equivalence to an arbitrary prior mean, a crucial aspect of formulating a well-specified model. To address the first limitation, we introduce a regularizer into the neural network's training objective, formally showing its correspondence to incorporating observation noise into the NTK-GP model. To address the second, we introduce a \textit{shifted network} that enables arbitrary prior mean functions. This approach allows us to perform gradient descent on a single neural network, without expensive ensembling or kernel matrix inversion. Our theoretical insights are validated empirically, with experiments exploring different values of observation noise and network architectures.