STENCIL: Submodular Mutual Information Based Weak Supervision for Cold-Start Active Learning
This addresses the challenge of reducing annotation costs for NLP applications with pre-trained models, particularly in imbalanced cold-start scenarios, though it is incremental as it builds on existing active learning and weak supervision techniques.
The paper tackles the problem of class imbalance in cold-start active learning for text classification by introducing STENCIL, which uses submodular mutual information to select weakly labeled rare-class instances for annotation. The result is an overall accuracy improvement of 10%-18% and a rare-class F-1 score boost of 17%-40% over common methods.
As supervised fine-tuning of pre-trained models within NLP applications increases in popularity, larger corpora of annotated data are required, especially with increasing parameter counts in large language models. Active learning, which attempts to mine and annotate unlabeled instances to improve model performance maximally fast, is a common choice for reducing the annotation cost; however, most methods typically ignore class imbalance and either assume access to initial annotated data or require multiple rounds of active learning selection before improving rare classes. We present STENCIL, which utilizes a set of text exemplars and the recently proposed submodular mutual information to select a set of weakly labeled rare-class instances that are then strongly labeled by an annotator. We show that STENCIL improves overall accuracy by $10\%-18\%$ and rare-class F-1 score by $17\%-40\%$ on multiple text classification datasets over common active learning methods within the class-imbalanced cold-start setting.