Infinite Neural Operators: Gaussian processes on functions
This work provides a theoretical foundation for understanding the inductive biases of neural operators, potentially improving uncertainty quantification in operator learning methods, though it is incremental as it builds on existing connections for neural networks.
The authors extended the connection between infinitely wide neural networks and Gaussian processes to neural operators, showing conditions under which arbitrary-depth neural operators with Gaussian convolution kernels converge to function-valued Gaussian processes, and computed their covariance functions and posteriors for regression tasks like PDE solution operators.
A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and transformers) induce Gaussian process (GP) priors over their outputs. These relationships provide both an accurate characterization of the prior predictive distribution and enable the use of GP machinery to improve the uncertainty quantification of deep neural networks. In this work, we extend this connection to neural operators (NOs), a class of models designed to learn mappings between function spaces. Specifically, we show conditions for when arbitrary-depth NOs with Gaussian-distributed convolution kernels converge to function-valued GPs. Based on this result, we show how to compute the covariance functions of these NO-GPs for two NO parametrizations, including the popular Fourier neural operator (FNO). With this, we compute the posteriors of these GPs in regression scenarios, including PDE solution operators. This work is an important step towards uncovering the inductive biases of current FNO architectures and opens a path to incorporate novel inductive biases for use in kernel-based operator learning methods.