Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?
This work addresses the challenge for practitioners in determining optimal training recipes for LLM-based embedding models by providing empirical insights into design effectiveness across different tasks.
The study investigates effective pooling and attention designs for LLM-based embedding models by conducting large-scale experiments with consistent training data and base models, finding that bidirectional attention and trainable pooling excel in text similarity and retrieval tasks, while simpler methods like EOS-last token pooling perform comparably in clustering and classification tasks, and proposes a Multi-Layers Trainable Pooling method that statistically outperforms existing methods in similarity and retrieval tasks.
The significant advancements of Large Language Models (LLMs) in generative tasks have led to a growing body of work exploring LLM-based embedding models. While these models, employing different pooling and attention strategies, have achieved state-of-the-art performance on public embedding benchmarks, questions still arise about what constitutes an effective design for LLM-based embedding models. However, these models are often trained on different datasets, using different LLM base models or training settings. Moreover, evaluations on public embedding benchmarks often fail to report statistical significance, making it difficult to determine which designs truly contribute to final performance. This complicates the process for practitioners seeking optimal training recipes for LLM-based embedding models. In this study, we conduct a large-scale experiment by training a series of LLM-based embedding models using the same training data and base model but differing in their pooling and attention strategies. The results show that there is no one-size-fits-all solution: while bidirectional attention and an additional trainable pooling layer outperform in text similarity and information retrieval tasks, they do not significantly surpass simpler designs like EOS-last token pooling and default causal attention in clustering and classification tasks. Furthermore, we propose a new pooling strategy, Multi-Layers Trainable Pooling, which transforms the outputs of all hidden layers, rather than just the last layer, using a cross-attention network. This method proves to be statistically superior in text similarity and retrieval tasks compared to existing pooling methods. Overall, this paper sheds light on effective training strategies for LLM-based embedding models.