Training neural network ensembles via trajectory sampling
This work addresses the training of neural network ensembles for machine learning practitioners, offering a novel approach but is incremental as it builds on existing ensemble and trajectory sampling techniques.
The authors tackled the problem of training neural network ensembles by defining them in terms of parameter trajectories under diffusive dynamics and biasing these trajectories towards low loss using counting fields, demonstrating viability on simple supervised learning tasks.
In machine learning, there is renewed interest in neural network ensembles (NNEs), whereby predictions are obtained as an aggregate from a diverse set of smaller models, rather than from a single larger model. Here, we show how to define and train a NNE using techniques from the study of rare trajectories in stochastic systems. We define an NNE in terms of the trajectory of the model parameters under a simple, and discrete in time, diffusive dynamics, and train the NNE by biasing these trajectories towards a small time-integrated loss, as controlled by appropriate counting fields which act as hyperparameters. We demonstrate the viability of this technique on a range of simple supervised learning tasks. We discuss potential advantages of our trajectory sampling approach compared with more conventional gradient based methods.