Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries
This work addresses the need for more effective ensemble methods in machine learning, particularly for large datasets like ImageNet, by offering a novel approach to hypothesis exploration, though it is incremental in advancing ensemble techniques.
The paper tackles the problem of improving diversity and generalization in deep ensembles by introducing Multi-Symmetry Ensembles (MSE), which capture multiple hypotheses along symmetry axes instead of relying on stochastic perturbations, resulting in enhanced classification performance, uncertainty quantification, and generalization across transfer tasks.
Deep ensembles (DE) have been successful in improving model performance by learning diverse members via the stochasticity of random initialization. While recent works have attempted to promote further diversity in DE via hyperparameters or regularizing loss functions, these methods primarily still rely on a stochastic approach to explore the hypothesis space. In this work, we present Multi-Symmetry Ensembles (MSE), a framework for constructing diverse ensembles by capturing the multiplicity of hypotheses along symmetry axes, which explore the hypothesis space beyond stochastic perturbations of model weights and hyperparameters. We leverage recent advances in contrastive representation learning to create models that separately capture opposing hypotheses of invariant and equivariant functional classes and present a simple ensembling approach to efficiently combine appropriate hypotheses for a given task. We show that MSE effectively captures the multiplicity of conflicting hypotheses that is often required in large, diverse datasets like ImageNet. As a result of their inherent diversity, MSE improves classification performance, uncertainty quantification, and generalization across a series of transfer tasks.