In-Context Learning for Text Classification with Many Labels
This addresses the problem of limited context windows in large language models for researchers and practitioners working on multi-label text classification, representing an incremental improvement over existing methods.
The paper tackled the challenge of in-context learning for text classification with many labels by using a dense retrieval model to bypass context window limitations, achieving new state-of-the-art performance in few-shot settings on intent classification datasets and surpassing fine-tuned models in some sentiment classification cases.
In-context learning (ICL) using large language models for tasks with many labels is challenging due to the limited context window, which makes it difficult to fit a sufficient number of examples in the prompt. In this paper, we use a pre-trained dense retrieval model to bypass this limitation, giving the model only a partial view of the full label space for each inference call. Testing with recent open-source LLMs (OPT, LLaMA), we set new state of the art performance in few-shot settings for three common intent classification datasets, with no finetuning. We also surpass fine-tuned performance on fine-grained sentiment classification in certain cases. We analyze the performance across number of in-context examples and different model scales, showing that larger models are necessary to effectively and consistently make use of larger context lengths for ICL. By running several ablations, we analyze the model's use of: a) the similarity of the in-context examples to the current input, b) the semantic content of the class names, and c) the correct correspondence between examples and labels. We demonstrate that all three are needed to varying degrees depending on the domain, contrary to certain recent works.