Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
This addresses the need for better intent classification in task-oriented dialogue systems with limited annotations, representing a strong specific gain rather than a broad breakthrough.
The paper tackles the problem of poor generalization in intent classification models when training data is limited, by proposing a pre-training method using contrastive learning with intent pseudo-labels, resulting in up to 5.4% and 4.0% higher accuracy in zero- and one-shot settings compared to previous state-of-the-art.
Intent classification (IC) plays an important role in task-oriented dialogue systems. However, IC models often generalize poorly when training without sufficient annotated examples for each user intent. We propose a novel pre-training method for text encoders that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks, reducing the need for manual annotations. By applying this pre-training strategy, we also introduce Pre-trained Intent-aware Encoder (PIE), which is designed to align encodings of utterances with their intent names. Specifically, we first train a tagger to identify key phrases within utterances that are crucial for interpreting intents. We then use these extracted phrases to create examples for pre-training a text encoder in a contrastive manner. As a result, our PIE model achieves up to 5.4% and 4.0% higher accuracy than the previous state-of-the-art text encoder for the N-way zero- and one-shot settings on four IC datasets.