Make (Nearly) Every Neural Network Better: Generating Neural Network Ensembles by Weight Parameter Resampling
This addresses the issue of computational cost in training for practitioners, though it is incremental as it builds on existing ensemble techniques.
The paper tackles the problem of improving neural network performance with minimal additional training by proposing a method to generate ensembles through weight parameter resampling, achieving performance gains on models like those pre-trained on ImageNet.
Deep Neural Networks (DNNs) have become increasingly popular in computer vision, natural language processing, and other areas. However, training and fine-tuning a deep learning model is computationally intensive and time-consuming. We propose a new method to improve the performance of nearly every model including pre-trained models. The proposed method uses an ensemble approach where the networks in the ensemble are constructed by reassigning model parameter values based on the probabilistic distribution of these parameters, calculated towards the end of the training process. For pre-trained models, this approach results in an additional training step (usually less than one epoch). We perform a variety of analysis using the MNIST dataset and validate the approach with a number of DNN models using pre-trained models on the ImageNet dataset.