Are We Really Making Much Progress in Text Classification? A Comparative Review
This work addresses the problem of evaluating progress in text classification for researchers and practitioners, highlighting that many advances are incremental and that robust comparisons are lacking.
The paper reviews text classification methods and finds that while newer models like graph-based and generative language models are emerging, encoder-only pre-trained models like BERT remain state-of-the-art, with simpler models such as logistic regression and trigram-based SVMs sometimes outperforming newer techniques.
We analyze various methods for single-label and multi-label text classification across well-known datasets, categorizing them into bag-of-words, sequence-based, graph-based, and hierarchical approaches. Despite the surge in methods like graph-based models, encoder-only pre-trained language models, notably BERT, remain state-of-the-art. However, recent findings suggest simpler models like logistic regression and trigram-based SVMs outperform newer techniques. While decoder-only generative language models show promise in learning with limited data, they lag behind encoder-only models in performance. We emphasize the superiority of discriminative language models like BERT over generative models for supervised tasks. Additionally, we highlight the literature's lack of robustness in method comparisons, particularly concerning basic hyperparameter optimizations like learning rate in fine-tuning encoder-only language models. Data availability: The source code is available at https://github.com/drndr/multilabel-text-clf All datasets used for our experiments are publicly available except the NYT dataset.