What's the Magic Word? A Control Theory of LLM Prompting
This provides a foundational perspective for enhancing language model capabilities by mathematically analyzing how input sequences steer outputs, addressing a poorly understood aspect of LLM deployment.
The authors tackled the problem of mathematically understanding prompt engineering in LLMs by formalizing them as discrete stochastic dynamical systems and analyzing controllability through control theory, finding that short prompts can reach the correct next token at least 97% of the time and dramatically alter output probabilities.
Prompt engineering is crucial for deploying LLMs but is poorly understood mathematically. We formalize LLM systems as a class of discrete stochastic dynamical systems to explore prompt engineering through the lens of control theory. We offer a mathematical analysis of the limitations on the controllability of self-attention as a function of the singular values of the parameter matrices. We present complementary empirical results on the controllability of a panel of LLMs, including Falcon-7b, Llama-7b, and Falcon-40b. Given initial state $\mathbf x_0$ from Wikitext and prompts of length $k \leq 10$ tokens, we find that the "correct" next token is reachable at least 97% of the time, and that the top 75 most likely next tokens are reachable at least 85% of the time. Intriguingly, short prompt sequences can dramatically alter the likelihood of specific outputs, even making the least likely tokens become the most likely ones. This control-theoretic analysis of LLMs demonstrates the significant and poorly understood role of input sequences in steering output probabilities, offering a foundational perspective for enhancing language model system capabilities.