Deep Combinatorial Aggregation
This work addresses uncertainty-aware learning for image classification tasks, offering incremental improvements over existing methods like deep ensemble and stochastic weight averaging.
The paper tackles the problem of poor uncertainty estimation in neural networks by introducing deep combinatorial aggregation (DCA), a generalization of deep ensemble, which improves predictive performance and uncertainty estimation, with coarse-grain DCAs outperforming deep ensemble and fine-grain DCAs matching or exceeding baselines like stochastic weight averaging without custom training schedules.
Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue. This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks. In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA). DCA creates multiple instances of network components and aggregates their combinations to produce diversified model proposals and predictions. DCA components can be defined at different levels of granularity. And we discovered that coarse-grain DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation. For fine-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training. It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers. Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA. We experiment on in-domain, distributional shift, and out-of-distribution image classification tasks, and empirically confirm the effectiveness of DCWA and DCA approaches.