Minibatch training of neural network ensembles via trajectory sampling
This work addresses the scalability issue in training neural network ensembles for machine learning practitioners, though it is incremental as it adapts existing minibatch methods to a specific context.
The paper tackles the problem of efficiently training neural network ensembles (NNEs) by introducing a minibatch approach via trajectory sampling, achieving a computational improvement of two orders of magnitude on MNIST dataset classification.
Most iterative neural network training methods use estimates of the loss function over small random subsets (or minibatches) of the data to update the parameters, which aid in decoupling the training time from the (often very large) size of the training datasets. Here, we show that a minibatch approach can also be used to train neural network ensembles (NNEs) via trajectory methods in a highly efficient manner. We illustrate this approach by training NNEs to classify images in the MNIST datasets. This method gives an improvement to the training times, allowing it to scale as the ratio of the size of the dataset to that of the average minibatch size which, in the case of MNIST, gives a computational improvement typically of two orders of magnitude. We highlight the advantage of using longer trajectories to represent NNEs, both for improved accuracy in inference and reduced update cost in terms of the samples needed in minibatch updates.