Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce
This work provides insights into the expressiveness of language models, which is foundational for researchers in AI and ML, though it is incremental as it builds on existing methods for prompt tuning.
The paper tackled the problem of understanding what probability distributions language models can produce by attempting to find prompts that induce specific target distributions, finding that distributions with extreme entropy or outlier tokens are easier to approximate, and LM-generated targets are more accessible than random ones.
Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. In this work, we attempt to systematically understand the probability distributions that LMs can produce, showing that some distributions are significantly harder to elicit than others. Specifically, for any target next-token distribution over the vocabulary, we attempt to find a prompt that induces the LM to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. We find that (1) in general, distributions with very low or very high entropy are easier to approximate than those with moderate entropy; (2) among distributions with the same entropy, those containing ''outlier tokens'' are easier to approximate; (3) target distributions generated by LMs -- even LMs with different tokenizers -- are easier to approximate than randomly chosen targets. These results offer insights into the expressiveness of LMs and the challenges of using them as probability distribution proposers.