Efficient Text Classification with Conformal In-Context Learning
This work addresses efficiency and scalability issues in text classification for NLP practitioners, but it is incremental as it builds on existing CICLe framework with broader evaluation.
The paper tackles the problem of high computational cost and prompt design dependency in LLM-based text classification by evaluating Conformal In-Context Learning (CICLe), which integrates a lightweight base classifier with Conformal Prediction to reduce candidate classes. The results show that CICLe improves over base classifiers and few-shot baselines in sufficient data, reduces shots and prompt length by up to 34.45% and 25.16%, and is advantageous for imbalanced tasks.
Large Language Models (LLMs) demonstrate strong in-context learning abilities, yet their effectiveness in text classification depends heavily on prompt design and incurs substantial computational cost. Conformal In-Context Learning (CICLe) has been proposed as a resource-efficient framework that integrates a lightweight base classifier with Conformal Prediction to guide LLM prompting by adaptively reducing the set of candidate classes. However, its broader applicability and efficiency benefits beyond a single domain have not yet been systematically explored. In this paper, we present a comprehensive evaluation of CICLe across diverse NLP classification benchmarks. The results show that CICLe consistently improves over its base classifier and outperforms few-shot prompting baselines when the sample size is sufficient for training the base classifier, and performs comparably in low-data regimes. In terms of efficiency, CICLe reduces the number of shots and prompt length by up to 34.45% and 25.16%, respectively, and enables the use of smaller models with competitive performance. CICLe is furthermore particularly advantageous for text classification tasks with high class imbalance. These findings highlight CICLe as a practical and scalable approach for efficient text classification, combining the robustness of traditional classifiers with the adaptability of LLMs, and achieving substantial gains in data and computational efficiency.