Mafin: Enhancing Black-Box Embeddings with Model Augmented Fine-Tuning
This addresses the need for domain-specific fine-tuning in retrieval-augmented generation systems, particularly when embeddings are from black-box models, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of sub-optimal performance of black-box embedding models in specific domains for retrieval-augmented generation by introducing Mafin, which fine-tunes these models with a small augmented component, resulting in significant performance enhancements validated on labeled and unlabeled datasets.
Retrieval Augmented Generation (RAG) has emerged as an effective solution for mitigating hallucinations in Large Language Models (LLMs). The retrieval stage in RAG typically involves a pre-trained embedding model, which converts queries and passages into vectors to capture their semantics. However, a standard pre-trained embedding model may exhibit sub-optimal performance when applied to specific domain knowledge, necessitating fine-tuning. This paper addresses scenarios where the embeddings are only available from a black-box model. We introduce Model augmented fine-tuning (Mafin) -- a novel approach for fine-tuning a black-box embedding model by augmenting it with a trainable embedding model. Our results demonstrate that Mafin significantly enhances the performance of the black-box embeddings by only requiring the training of a small augmented model. We validate the effectiveness of our method on both labeled and unlabeled datasets, illustrating its broad applicability and efficiency.