Layer-Aware Embedding Fusion for LLMs in Text Classifications
This work addresses a systematic gap in embedding fusion for LLMs in text classification, offering incremental improvements for NLP practitioners by optimizing layer selection and model combination.
The study tackled the problem of selecting optimal layers and fusion strategies for embedding fusion in LLMs for text classification, showing that critical layers vary by dataset and combining embeddings from multiple models can enhance performance without fine-tuning, with experiments on four datasets (SST-2, MR, R8, R52) demonstrating improved results.
Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real world feasibility of embedding fusion. Future work will explore multilingual and domain specific datasets, as well as techniques for automating layer selection, to improve both performance and scalability.