From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models
This addresses a gap in NLP for educational applications, but the results are incremental as they show LLMs underperform compared to existing models like BERT.
The study investigated the ability of Large Language Models (LLMs) to predict Chinese classifiers, finding that LLMs perform worse than BERT, even with fine-tuning, and that prediction benefits from information about the following noun.
Classifiers are an important and defining feature of the Chinese language, and their correct prediction is key to numerous educational applications. Yet, whether the most popular Large Language Models (LLMs) possess proper knowledge the Chinese classifiers is an issue that has largely remain unexplored in the Natural Language Processing (NLP) literature. To address such a question, we employ various masking strategies to evaluate the LLMs' intrinsic ability, the contribution of different sentence elements, and the working of the attention mechanisms during prediction. Besides, we explore fine-tuning for LLMs to enhance the classifier performance. Our findings reveal that LLMs perform worse than BERT, even with fine-tuning. The prediction, as expected, greatly benefits from the information about the following noun, which also explains the advantage of models with a bidirectional attention mechanism such as BERT.