Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study
This addresses the challenge of domain-specific text classification for researchers and practitioners in scientific fields, but it is incremental as it applies existing fine-tuning methods to new data.
The study tackled the problem of automated text classification for scientific texts by fine-tuning four large language models (BERT, SciBERT, BioBERT, BlueBERT) on datasets from WoS-46985, finding that domain-specific models like SciBERT consistently outperformed general-purpose ones in abstract-based and keyword-based tasks.
The exponential growth of online textual content across diverse domains has necessitated advanced methods for automated text classification. Large Language Models (LLMs) based on transformer architectures have shown significant success in this area, particularly in natural language processing (NLP) tasks. However, general-purpose LLMs often struggle with domain-specific content, such as scientific texts, due to unique challenges like specialized vocabulary and imbalanced data. In this study, we fine-tune four state-of-the-art LLMs BERT, SciBERT, BioBERT, and BlueBERT on three datasets derived from the WoS-46985 dataset to evaluate their performance in scientific text classification. Our experiments reveal that domain-specific models, particularly SciBERT, consistently outperform general-purpose models in both abstract-based and keyword-based classification tasks. Additionally, we compare our achieved results with those reported in the literature for deep learning models, further highlighting the advantages of LLMs, especially when utilized in specific domains. The findings emphasize the importance of domain-specific adaptations for LLMs to enhance their effectiveness in specialized text classification tasks.