A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?
This work addresses the problem of detecting LLM-generated synthetic texts for researchers and practitioners, offering incremental insights into fractal properties as a detection method.
The study investigated whether large language models (LLMs) can replicate the fractal complexity of natural language, finding that LLM outputs vary widely in fractal parameters compared to the narrow range in human texts, enabling detection of synthetic texts with robustness across architectures like Gemini and Mistral.
Language exhibits a fractal structure in its information-theoretic complexity (i.e. bits per token), with self-similarity across scales and long-range dependence (LRD). In this work, we investigate whether large language models (LLMs) can replicate such fractal characteristics and identify conditions-such as temperature setting and prompting method-under which they may fail. Moreover, we find that the fractal parameters observed in natural language are contained within a narrow range, whereas those of LLMs' output vary widely, suggesting that fractal parameters might prove helpful in detecting a non-trivial portion of LLM-generated texts. Notably, these findings, and many others reported in this work, are robust to the choice of the architecture; e.g. Gemini 1.0 Pro, Mistral-7B and Gemma-2B. We also release a dataset comprising of over 240,000 articles generated by various LLMs (both pretrained and instruction-tuned) with different decoding temperatures and prompting methods, along with their corresponding human-generated texts. We hope that this work highlights the complex interplay between fractal properties, prompting, and statistical mimicry in LLMs, offering insights for generating, evaluating and detecting synthetic texts.