Critical Phase Transition in Large Language Models
This work provides a novel analogy between LLMs and natural phenomena, potentially offering insights into model behaviors for researchers, though it is incremental in applying phase transition concepts to AI.
The paper investigates whether qualitative changes in Large Language Models (LLMs) when varying the temperature parameter constitute phase transitions, finding that statistical quantities diverge at a critical point between low-temperature repetitive structures and high-temperature incomprehensible sentences, with behaviors similar to natural languages.
Large Language Models (LLMs) have demonstrated impressive performance. To understand their behaviors, we need to consider the fact that LLMs sometimes show qualitative changes. The natural world also presents such changes called phase transitions, which are defined by singular, divergent statistical quantities. Therefore, an intriguing question is whether qualitative changes in LLMs are phase transitions. In this work, we have conducted extensive analysis on texts generated by LLMs and suggested that a phase transition occurs in LLMs when varying the temperature parameter. Specifically, statistical quantities have divergent properties just at the point between the low-temperature regime, where LLMs generate sentences with clear repetitive structures, and the high-temperature regime, where generated sentences are often incomprehensible. In addition, critical behaviors near the phase transition point, such as a power-law decay of correlation and slow convergence toward the stationary state, are similar to those in natural languages. Our results suggest a meaningful analogy between LLMs and natural phenomena.