On the probability-quality paradox in language generation
This addresses a fundamental issue in natural language generation for researchers and practitioners, offering a theoretical insight into decoding strategies, though it is incremental as it builds on existing observations.
The paper tackles the paradox where high-probability text from neural models is often unnatural, while lower-probability text is more human-like, by proposing an information-theoretic explanation that human-like language should have information content near the entropy of natural strings, with preliminary evidence showing high-quality text aligns significantly closer to entropy than expected by chance.
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decoding methods, i.e., those that produce high-probability text under the model, lead to unnatural language. On the other hand, the lower-probability text generated by stochastic methods is perceived as more human-like. In this note, we offer an explanation for this phenomenon by analyzing language generation through an information-theoretic lens. Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings. Further, we posit that language with substantially more (or less) information is undesirable. We provide preliminary empirical evidence in favor of this hypothesis; quality ratings of both human and machine-generated text -- covering multiple tasks and common decoding strategies -- suggest high-quality text has an information content significantly closer to the entropy than we would expect by chance.