Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression
This work provides a theoretical foundation for overparameterized nonlinear models, which is incremental as it extends existing Bayesian methods to address a known bottleneck in the field.
This paper tackles the problem of understanding overparameterization in nonlinear regression models by proposing a Bayesian framework with an adaptive prior that leverages the data's spectral structure. The result is a method that achieves consistent predictions and reliable uncertainty estimates, validated through simulations and real data.
The remarkable generalization performance of large-scale models has been challenging the conventional wisdom of the statistical learning theory. Although recent theoretical studies have shed light on this behavior in linear models and nonlinear classifiers, a comprehensive understanding of overparameterization in nonlinear regression models is still lacking. This study explores the predictive properties of overparameterized nonlinear regression within the Bayesian framework, extending the methodology of the adaptive prior considering the intrinsic spectral structure of the data. Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions, demonstrating the consistency in the predictions of the proposed approach. Moreover, the Bayesian framework enables uncertainty estimation of the predictions. The proposed method was validated via numerical simulations and a real data application, showing its ability to achieve accurate predictions and reliable uncertainty estimates. This work provides a theoretical understanding of the advantages of overparameterization and a principled Bayesian approach to large nonlinear models.