CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora
This addresses the issue of costly human annotation for domain-specific retrieval, enabling cheaper RAG deployments, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of dense embedding models degrading on specialized corpora by introducing CustomIR, an unsupervised framework that uses synthetic query-document pairs for fine-tuning, resulting in up to 2.3 points improvement in Recall@10 on enterprise datasets.
Dense embedding models have become critical for modern information retrieval, particularly in RAG pipelines, but their performance often degrades when applied to specialized corpora outside their pre-training distribution. To address thi we introduce CustomIR, a framework for unsupervised adaptation of pre-trained language embedding models to domain-specific corpora using synthetically generated query-document pairs. CustomIR leverages large language models (LLMs) to create diverse queries grounded in a known target corpus, paired with LLM-verified hard negatives, eliminating the need for costly human annotation. Experiments on enterprise email and messaging datasets show that CustomIR consistently improves retrieval effectiveness with small models gaining up to 2.3 points in Recall@10. This performance increase allows these small models to rival the performance of much larger alternatives, allowing for cheaper RAG deployments. These results highlight that targeted synthetic fine-tuning offers a scalable and cost-efficient strategy for increasing domain-specific performance.