SGPT: GPT Sentence Embeddings for Semantic Search
This work addresses the challenge for organizations in NLP fields that need efficient and high-performing models for semantic search, offering a novel approach that reduces the need for separate models.
The paper tackles the problem of using large decoder transformers for semantic search and sentence embeddings, achieving a 7% improvement over previous best sentence embeddings and outperforming a concurrent method with 175 billion parameters on the BEIR search benchmark.
Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.