PACE: Prune-And-Compress Ensemble Models
For practitioners deploying ensemble models, PACE offers a way to reduce model size without sacrificing accuracy, with controlled faithfulness to the original ensemble.
PACE introduces a two-phase framework that first generates diverse learners to enrich an ensemble, then prunes it, achieving better performance than prior pruning and compression methods while providing principled faithfulness guarantees.
Ensemble models achieve state-of-the-art performance on prediction tasks, but usually require aggregating a large number of weak learners. This can hinder deployment, interpretability, and downstream tasks such as robustness verification. Remedies to this issue fall into two main camps: pruning, which discards redundant learners, and compression, which generates new ones from scratch. We introduce PACE, a framework that interleaves these paradigms in a two-phase strategy. First, new learners are actively generated via a theoretically grounded procedure to enhance the diversity of the initial ensemble. When no more relevant learners can be found, a second phase of pruning is performed on this enriched ensemble. During both operations, PACE allows fine control on the faithfulness to the original ensemble. Experiments show that our method outperforms prior pruning and compression methods while offering principled control of faithfulness guarantees.