LLMs Know More Than Words: A Genre Study with Syntax, Metaphor & Phonetics
This work addresses the problem of understanding LLMs' linguistic capabilities for researchers in natural language processing, though it is incremental as it builds on existing genre classification methods.
The study investigated whether large language models (LLMs) can capture deeper linguistic properties like syntax, metaphor, and phonetics from raw text, using a multilingual genre classification dataset from Project Gutenberg. Results showed that LLMs can learn these features, but their contributions varied across tasks, highlighting the need for incorporating complex linguistic signals in training.
Large language models (LLMs) demonstrate remarkable potential across diverse language related tasks, yet whether they capture deeper linguistic properties, such as syntactic structure, phonetic cues, and metrical patterns from raw text remains unclear. To analysis whether LLMs can learn these features effectively and apply them to important nature language related tasks, we introduce a novel multilingual genre classification dataset derived from Project Gutenberg, a large-scale digital library offering free access to thousands of public domain literary works, comprising thousands of sentences per binary task (poetry vs. novel;drama vs. poetry;drama vs. novel) in six languages (English, French, German, Italian, Spanish, and Portuguese). We augment each with three explicit linguistic feature sets (syntactic tree structures, metaphor counts, and phonetic metrics) to evaluate their impact on classification performance. Experiments demonstrate that although LLM classifiers can learn latent linguistic structures either from raw text or from explicitly provided features, different features contribute unevenly across tasks, which underscores the importance of incorporating more complex linguistic signals during model training.