Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?
This work addresses the problem of model selection for text classification tasks, advocating for a task-driven approach rather than relying solely on LLMs, though it is incremental as it builds on existing comparisons.
This study tackled the problem of whether BERT-like models still outperform LLMs in text classification by comparing methods across six high-difficulty datasets, finding that BERT-like models often perform better, especially in pattern-driven tasks, while LLMs excel in tasks requiring deep semantics or world knowledge.
The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.