Revisiting Distance Metric Learning for Few-Shot Natural Language Classification
This work addresses few-shot learning challenges in NLP classification, offering incremental improvements for language model fine-tuning.
The paper tackled the problem of improving few-shot natural language classification by applying distance metric learning (DML) to fine-tune RoBERTa models, finding that combining categorical cross-entropy loss with ProxyAnchor Loss boosts performance by an average of 3.27 percentage points, with gains up to 10.38 percentage points.
Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classification tasks under few-shot learning settings. We investigated several DML loss functions in training RoBERTa language models on known SentEval Transfer Tasks datasets. We also analyzed the possibility of using proxy-based DML losses during model inference. Our systematic experiments have shown that under few-shot learning settings, particularly proxy-based DML losses can positively affect the fine-tuning and inference of a supervised language model. Models tuned with a combination of CCE (categorical cross-entropy loss) and ProxyAnchor Loss have, on average, the best performance and outperform models with only CCE by about 3.27 percentage points -- up to 10.38 percentage points depending on the training dataset.