Bagging-Based Model Merging for Robust General Text Embeddings
This work provides a method for improving the robustness and incremental adaptability of general-purpose text embedding models, which is important for developers and researchers working with evolving NLP and information retrieval applications.
This paper investigates multi-task training strategies for text embeddings, finding that simple batch-level shuffling is effective but suffers from suboptimal out-of-domain generalization and high retraining costs for incremental learning. To address these, they propose BOOM, a bagging-based model merging approach that trains multiple models on sampled subsets and merges them, improving both in-domain and out-of-domain performance while reducing incremental learning costs.
General-purpose text embedding models underpin a wide range of NLP and information retrieval applications, and are typically trained on large-scale multi-task corpora to encourage broad generalization. However, it remains unclear how different multi-task training strategies compare in practice, and how to efficiently adapt embedding models as new domains and data types continually emerge. In this work, we present a systematic study of multi-task training for text embeddings from two perspectives: data scheduling and model merging. We compare batch-level shuffling, sequential training variants, two-stage training, and multiple merging granularities, and find that simple batch-level shuffling consistently yields the strongest overall performance, suggesting that task conflicts are limited and training datasets are largely complementary. Despite its effectiveness, batch-level shuffling exhibits two practical limitations: suboptimal out-of-domain (OOD) generalization and poor suitability for incremental learning due to expensive full retraining. To address these issues, we propose Bagging-based rObust mOdel Merging (BOOM), which trains multiple embedding models on sampled subsets and merges them into a single model, improving robustness while retaining single-model inference efficiency. Moreover, BOOM naturally supports efficient incremental updates by training lightweight update models on new data with a small historical subset and merging them into the existing model. Experiments across diverse embedding benchmarks demonstrate that BOOM consistently improves both in-domain and OOD performance over full-corpus batch-level shuffling, while substantially reducing training cost in incremental learning settings.