Characterizing Tradeoffs in Language Model Decoding with Informational Interpretations
This work addresses the challenge of designing and interpreting decoder algorithms for language models, which is incremental as it provides a new theoretical perspective on existing methods.
The authors tackled the problem of understanding and arbitrating tradeoffs in language model decoding by proposing a theoretical framework that uses dynamic programming and information theory to interpret decoder algorithms in terms of action-state value functions, showing that these algorithms optimize for sensibleness, diversity, and attribution.
We propose a theoretical framework for formulating language model decoder algorithms with dynamic programming and information theory. With dynamic programming, we lift the design of decoder algorithms from the logit space to the action-state value function space, and show that the decoding algorithms are consequences of optimizing the action-state value functions. Each component in the action-state value function space has an information theoretical interpretation. With the lifting and interpretation, it becomes evident what the decoder algorithm is optimized for, and hence facilitating the arbitration of the tradeoffs in sensibleness, diversity, and attribution.