MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity
This work addresses a computational bottleneck in uncertainty quantification for Bayesian neural networks, offering an incremental improvement for researchers and practitioners in machine learning.
The paper tackled the computational inefficiency of moment propagation in Bayesian neural networks by proposing MP-GELU, a new nonlinear function that enables faster analytical computation of moments, resulting in higher prediction accuracy, better uncertainty quality, and faster execution compared to ReLU-based BNNs in regression tasks.
Bayesian neural networks (BNNs) have been an important framework in the study of uncertainty quantification. Deterministic variational inference, one of the inference methods, utilizes moment propagation to compute the predictive distributions and objective functions. Unfortunately, deriving the moments requires computationally expensive Taylor expansion in nonlinear functions, such as a rectified linear unit (ReLU) or a sigmoid function. Therefore, a new nonlinear function that realizes faster moment propagation than conventional functions is required. In this paper, we propose a novel nonlinear function named moment propagating-Gaussian error linear unit (MP-GELU) that enables the fast derivation of first and second moments in BNNs. MP-GELU enables the analytical computation of moments by applying nonlinearity to the input statistics, thereby reducing the computationally expensive calculations required for nonlinear functions. In empirical experiments on regression tasks, we observed that the proposed MP-GELU provides higher prediction accuracy and better quality of uncertainty with faster execution than those of ReLU-based BNNs.