IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classification
This addresses the challenge of few-label text classification for NLP practitioners, though it is incremental as it builds on existing intermediate training approaches.
The paper tackles the cold-start problem in text classification with limited labels by introducing IDoFew, a two-stage intermediate clustering method that reduces errors in generating pseudo-labels for fine-tuning, resulting in significant improvements over comparative models.
Language models such as Bidirectional Encoder Representations from Transformers (BERT) have been very effective in various Natural Language Processing (NLP) and text mining tasks including text classification. However, some tasks still pose challenges for these models, including text classification with limited labels. This can result in a cold-start problem. Although some approaches have attempted to address this problem through single-stage clustering as an intermediate training step coupled with a pre-trained language model, which generates pseudo-labels to improve classification, these methods are often error-prone due to the limitations of the clustering algorithms. To overcome this, we have developed a novel two-stage intermediate clustering with subsequent fine-tuning that models the pseudo-labels reliably, resulting in reduced prediction errors. The key novelty in our model, IDoFew, is that the two-stage clustering coupled with two different clustering algorithms helps exploit the advantages of the complementary algorithms that reduce the errors in generating reliable pseudo-labels for fine-tuning. Our approach has shown significant improvements compared to strong comparative models.