Intra-Ensemble in Neural Networks
This work addresses the resource efficiency problem for machine learning practitioners by offering an incremental improvement over traditional ensemble techniques in deep learning.
The authors tackled the problem of high resource costs in training multiple independent neural networks for ensemble methods by proposing Intra-Ensemble, a strategy that trains several sub-networks simultaneously within a single network using stochastic channel recombination, resulting in enhanced ensemble performance with marginal additional parameters.
Improving model performance is always the key problem in machine learning including deep learning. However, stand-alone neural networks always suffer from marginal effect when stacking more layers. At the same time, ensemble is an useful technique to further enhance model performance. Nevertheless, training several independent deep neural networks for ensemble costs multiple resources. If so, is it possible to utilize ensemble in only one neural network? In this work, we propose Intra-Ensemble, an end-to-end ensemble strategy with stochastic channel recombination operations to train several sub-networks simultaneously within one neural network. Additional parameter size is marginal since the majority of parameters are mutually shared. Meanwhile, stochastic channel recombination significantly increases the diversity of sub-networks, which finally enhances ensemble performance. Extensive experiments and ablation studies prove the applicability of intra-ensemble on various kinds of datasets and network architectures.