Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
This identifies a critical bottleneck for practitioners using HMC in Bayesian deep learning, making it incremental by analyzing a known issue in a specific context.
The paper tackled the inefficiency of Hamiltonian Monte Carlo (HMC) for Bayesian inference in ReLU neural networks, showing that non-differentiability leads to a large local error rate of Ω(ε) instead of O(ε³), resulting in higher rejection rates and inefficiency, as verified by simulations and real-world experiments.
We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $Ω(ε)$ rather than the classical error rate of $O(ε^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.