States of LLM-generated Texts and Phase Transitions between them
This work provides insights into text generation dynamics for researchers in NLP and physics, but it is incremental as it builds on known autocorrelation differences.
The study investigated how the temperature parameter in LLMs affects text generation, showing that it can produce text classified as solid, critical state, or gas based on autocorrelation decay patterns.
It is known for some time that autocorrelations of words in human-written texts decay according to a power law. Recent works have also shown that the autocorrelations decay in texts generated by LLMs is qualitatively different from the literary texts. Solid state physics tie the autocorrelations decay laws to the states of matter. In this work, we empirically demonstrate that, depending on the temperature parameter, LLMs can generate text that can be classified as solid, critical state or gas.