Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
This is an incremental improvement for machine learning practitioners seeking better ensemble performance without training multiple models from scratch.
The paper tackled the problem of limited diversity in ensembles from a single pretrained model by finetuning different subnetworks, showing that this approach outperformed standard ensembles on some tasks.
Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.