TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
This work addresses the problem of LLMs' limited ability to handle varied linguistic queries for users in expert domains, though it is incremental as it builds on existing analysis methods.
The study investigated how large language models (LLMs) perform on diverse linguistic query types beyond definitions, finding that task performance is highest for definition-type questions and lowest for exemplification-type questions, with LLMs paraphrasing more on popular knowledge and less on tail knowledge.
Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.