Making Pretrained Language Models Good Long-tailed Learners
This addresses long-tailed classification challenges in NLP, offering a method to improve performance on tail classes, though it is incremental as it builds on existing prompt-tuning techniques.
The paper investigates whether prompt-tuning, effective in few-shot classification, also works for long-tailed classification, where tail classes resemble few-shot scenarios, and finds that it makes pretrained language models good long-tailed learners, with analyses highlighting classifier structure and parameterization as key factors.
Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification. Good long-tailed learners can be abbreviated as Glee.