Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples
This work addresses the data annotation bottleneck for researchers and practitioners in NLP by enabling efficient sentence embedding training with minimal manual effort, though it is incremental as it builds on existing models like PromptEOL.
The paper tackles the problem of requiring large manually annotated datasets for fine-tuning sentence embedding models by automatically generating a natural language inference dataset using few-shot learning with an LLM. The result is improved performance on semantic textual similarity tasks, outperforming existing models in settings without large manual annotations.
Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for fine-tuning. We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for fine-tuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.