Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training
This work addresses the problem of data efficiency in intent classification for conversational AI systems, offering a simpler and potentially more effective approach that reduces reliance on external data, though it is incremental in optimizing existing methods.
The paper tackles few-shot intent classification by showing that direct fine-tuning of pre-trained language models on limited labeled data yields competitive results compared to continual pre-training, with performance gaps diminishing as data increases, achieving strong benchmarks with only two or more labeled samples per class.
We consider the task of few-shot intent detection, which involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data. The current approach to address this problem is through continual pre-training, i.e., fine-tuning pre-trained language models (PLMs) on external resources (e.g., conversational corpora, public intent detection datasets, or natural language understanding datasets) before using them as utterance encoders for training an intent classifier. In this paper, we show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected. Specifically, we find that directly fine-tuning PLMs on only a handful of labeled examples already yields decent results compared to methods that employ continual pre-training, and the performance gap diminishes rapidly as the number of labeled data increases. To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance. Comprehensive experiments on real-world benchmarks show that given only two or more labeled samples per class, direct fine-tuning outperforms many strong baselines that utilize external data sources for continual pre-training. The code can be found at https://github.com/hdzhang-code/DFTPlus.