The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification
This work addresses the challenge of text classification with few labeled examples, but it is incremental as it builds directly on prior research.
The authors tackled the problem of semi-supervised text classification with limited labeled data by extending an existing teacher-student model with unsupervised pre-training using objective masking, resulting in performance improvements across multiple datasets and languages.
We extend and study a semi-supervised model for text classification proposed earlier by Hatefi et al. for classification tasks in which document classes are described by a small number of gold-labeled examples, while the majority of training examples is unlabeled. The model leverages the teacher-student architecture of Meta Pseudo Labels in which a ''teacher'' generates labels for originally unlabeled training data to train the ''student'' and updates its own model iteratively based on the performance of the student on the gold-labeled portion of the data. We extend the original model of Hatefi et al. by an unsupervised pre-training phase based on objective masking, and conduct in-depth performance evaluations of the original model, our extension, and various independent baselines. Experiments are performed using three different datasets in two different languages (English and Swedish).