BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings
This work addresses the need for better semantic similarity measurements in natural language processing, though it appears incremental by modifying attention mechanisms in existing models.
The authors tackled the problem of improving sentence embeddings by incorporating backward dependencies into autoregressive large language models, achieving state-of-the-art performance on semantic textual similarity tasks.
Sentence embeddings are crucial in measuring semantic similarity. Most recent studies employed large language models (LLMs) to learn sentence embeddings. Existing LLMs mainly adopted autoregressive architecture without explicit backward dependency modeling. Therefore, we examined the effects of backward dependencies in LLMs for semantic similarity measurements. Concretely, we propose a novel model: backward dependency enhanced large language model (BeLLM). It learns sentence embeddings via transforming specific attention layers from uni- to bi-directional. We extensively experiment across various semantic textual similarity (STS) tasks and downstream applications. BeLLM achieves state-of-the-art performance in varying scenarios. It shows that auto-regressive LLMs benefit from backward dependencies for sentence embeddings.