DIVERSE: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration
This work addresses the need for efficient exploration of model multiplicity in machine learning, offering a method to find diverse high-performing models without full retraining, though it is incremental compared to baseline retraining approaches.
The paper tackles the problem of exploring the Rashomon set of deep neural networks, which are models with similar accuracy but different predictive behaviors, by proposing DIVERSE, a framework that uses FiLM layers and CMA-ES to generate diverse model variants without retraining, achieving competitive diversity at reduced computational cost across datasets like MNIST, PneumoniaMNIST, and CIFAR-10.
We propose DIVERSE, a framework for systematically exploring the Rashomon set of deep neural networks, the collection of models that match a reference model's accuracy while differing in their predictive behavior. DIVERSE augments a pretrained model with Feature-wise Linear Modulation (FiLM) layers and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search a latent modulation space, generating diverse model variants without retraining or gradient access. Across MNIST, PneumoniaMNIST, and CIFAR-10, DIVERSE uncovers multiple high-performing yet functionally distinct models. Our experiments show that DIVERSE offers a competitive and efficient exploration of the Rashomon set, making it feasible to construct diverse sets that maintain robustness and performance while supporting well-balanced model multiplicity. While retraining remains the baseline to generate Rashomon sets, DIVERSE achieves comparable diversity at reduced computational cost.