Better Call Claude: Can LLMs Detect Changes of Writing Style?
This addresses a challenging task in authorship analysis for researchers and practitioners, though it appears incremental as it benchmarks existing models on new datasets.
The paper tackled the problem of detecting changes in writing style at the sentence level using large language models (LLMs) in a zero-shot setting, finding that state-of-the-art LLMs are sensitive to such variations and outperform PAN competition baselines.
This article explores the zero-shot performance of state-of-the-art large language models (LLMs) on one of the most challenging tasks in authorship analysis: sentence-level style change detection. Benchmarking four LLMs on the official PAN~2024 and 2025 "Multi-Author Writing Style Analysis" datasets, we present several observations. First, state-of-the-art generative models are sensitive to variations in writing style - even at the granular level of individual sentences. Second, their accuracy establishes a challenging baseline for the task, outperforming suggested baselines of the PAN competition. Finally, we explore the influence of semantics on model predictions and present evidence suggesting that the latest generation of LLMs may be more sensitive to content-independent and purely stylistic signals than previously reported.