Reversible Jump MCMC Simulated Annealing for Neural Networks
This addresses model selection and optimization challenges in neural networks, though it appears incremental as it builds on existing MCMC and simulated annealing techniques.
The paper tackles the problem of local minima in optimizing radial basis function networks by proposing a reversible jump MCMC simulated annealing algorithm that maximizes the joint posterior distribution of network parameters and the number of basis functions, showing theoretical and empirical convergence to posterior modes.
We propose a novel reversible jump Markov chain Monte Carlo (MCMC) simulated annealing algorithm to optimize radial basis function (RBF) networks. This algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis functions. It performs a global search in the joint space of the parameters and number of parameters, thereby surmounting the problem of local minima. We also show that by calibrating a Bayesian model, we can obtain the classical AIC, BIC and MDL model selection criteria within a penalized likelihood framework. Finally, we show theoretically and empirically that the algorithm converges to the modes of the full posterior distribution in an efficient way.