Neural network with optimal neuron activation functions based on additive Gaussian process regression
This work addresses the computational cost and overfitting issues in neural networks for scientific applications like molecular modeling, though it is incremental as it builds on existing methods with a novel combination.
The authors tackled the problem of limited expressive power in feed-forward neural networks due to simple, uniform neuron activation functions by developing a method using additive Gaussian process regression to construct optimal, individual activation functions for each neuron, which outperformed a conventional neural network in high-accuracy regimes for fitting potential energy surfaces of water and formaldehyde without non-linear optimization.
Feed-forward neural networks (NN) are a staple machine learning method widely used in many areas of science and technology. While even a single-hidden layer NN is a universal approximator, its expressive power is limited by the use of simple neuron activation functions (such as sigmoid functions) that are typically the same for all neurons. More flexible neuron activation functions would allow using fewer neurons and layers and thereby save computational cost and improve expressive power. We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron. An approach is also introduced that avoids non-linear fitting of neural network parameters. The resulting method combines the advantage of robustness of a linear regression with the higher expressive power of a NN. We demonstrate the approach by fitting the potential energy surfaces of the water molecule and formaldehyde. Without requiring any non-linear optimization, the additive GPR based approach outperforms a conventional NN in the high accuracy regime, where a conventional NN suffers more from overfitting.