Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
This addresses the challenge of ensemble diversity for practitioners fine-tuning foundation models, though it appears incremental as it builds on existing Bayesian and distributional robustness approaches.
The paper tackles the problem of improving ensemble quality in fine-tuning foundation models by introducing Interactive Bayesian Distributional Robustness (IBDR), a Bayesian inference framework that models particle interactions to enhance diversity. Results show IBDR consistently outperforms baseline methods on the VTAB-1K benchmark and common reasoning language tasks.
We introduce Interactive Bayesian Distributional Robustness (IBDR), a novel Bayesian inference framework that allows modeling the interactions between particles, thereby enhancing ensemble quality through increased particle diversity. IBDR is grounded in a generalized theoretical framework that connects the distributional population loss with the approximate posterior, motivating a practical dual optimization procedure that enforces distributional robustness while fostering particle diversity. We evaluate IBDR's performance against various baseline methods using the VTAB-1K benchmark and the common reasoning language task. The results consistently show that IBDR outperforms these baselines, underscoring its effectiveness in real-world applications.