Variational Pretraining for Semi-supervised Text Classification
This addresses the challenge of effective text classification for users with constrained computational resources, though it is incremental as it builds on existing pretraining and semi-supervised techniques.
The paper tackles the problem of text classification with limited data and computing resources by introducing VAMPIRE, a lightweight pretraining framework that uses a variational autoencoder on in-domain, unlabeled data to generate features for downstream classifiers, showing competitive performance against more expensive methods in low-resource settings.
We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.