Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems
This addresses scalability issues for ensemble methods in large-scale deep learning, though it appears incremental as it builds on existing low precision techniques.
The paper tackles the challenge of scaling ensemble methods for large neural networks by proposing a training-free low precision ensembling approach, showing its effectiveness compared to existing methods.
While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.