Learning Neural Network Subspaces
This addresses the computational inefficiency of training multiple models for ensembling in machine learning, offering a more practical approach for practitioners.
The paper tackles the problem of efficiently finding diverse, high-accuracy neural network solutions by learning subspaces (lines, curves, simplexes) in a single training run, achieving ensemble-like performance without the cost of multiple runs and improving accuracy, calibration, and robustness over methods like Stochastic Weight Averaging.
Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.